The idea is that we can translate Console API calls directly to VT at
least as well as the current VtEngine setup can. For instance, a call
to `SetConsoleCursorPosition` clearly translates directly to a `CUP`
escape sequence. Effectively, instead of translating output
asynchronously in the renderer thread, we'll do it synchronously
right during the Console API call.
Most importantly, the this means that any VT output that an
application generates will now be given to the terminal unmodified.
Aside from reducing our project's complexity quite a bit and opening
the path towards various interesting work like sixels, Device Control
Strings, buffer snapshotting, synchronized updates, and more, it also
improves performance for mixed text output like enwik8.txt in conhost
to 1.3-2x and in Windows Terminal via ConPTY to roughly 20x.
This adds support for overlapped IO, because now that output cannot
be "skipped" anymore (VtEngine worked like a renderer after all)
it's become crucial to block conhost as little as possible.
⚠️ Intentionally unresolved changes/quirks:
* To force a delayed EOL wrap to wrap, `WriteCharsLegacy` emits a
`\r\n` if necessary. This breaks text reflow on window resize.
We cannot emit ` \r` the way readline does it, because this would
overwrite the first column in the next row with a whitespace.
The alternative is to read back the affected cell from the buffer
and emit that character and its attributes followed by a `\r`.
I chose to not do that, because buffer read-back is lossy (= UCS2).
Unless the window is resized, the difference is unnoticeable
and historically, conhost had no support for buffer reflow anyway.
* If `ENABLE_VIRTUAL_TERMINAL_PROCESSING` is set while
`DISABLE_NEWLINE_AUTO_RETURN` is reset, we'll blindly replace all
LF with CRLF. This may hypothetically break DCS sequences, but it's
the only way to do this without parsing the given VT string and
thus the only way we can achieve passthrough mode in the future.
* `ENABLE_WRAP_AT_EOL_OUTPUT` is translated to `DECAWM`.
Between Windows XP and Windows 11 21H2, `ENABLE_WRAP_AT_EOL_OUTPUT`
being reset would cause the cursor position to reset to wherever
a write started, _if_ the write, including expanded control chars,
was less than 100 characters long. If it was longer than that,
the cursor position would end up in an effectively random position.
After lengthy research I believe that this is a bug introduced in
Windows XP and that the original intention was for this mode to be
equivalent to `DECAWM`. This is compounded by MSDN's description
(emphasis mine):
> If this mode is disabled, the **last character** in the row is
> overwritten with any subsequent characters.
⚠️ Unresolved issues/quirks:
* Focus/Unfocus events are injected into the output stream without
checking whether the VT output is currently in a ground state.
This may break whatever VT sequence is currently ongoing.
This is an existing issue.
* `VtIo::Writer::WriteInfos` should properly verify the width of
each individual character.
* Using `SetConsoleActiveScreenBuffer` destroys surrogate pairs
and extended (VT) attributes. It could be translated to VT pages
in the long term.
* Similarly, `ScrollConsoleScreenBuffer` results in the same and
could be translated to `DECCRA` and `DECFRA` in the near term.
This is important because otherwise `vim` output may loose
its extended attributes during scrolling.
* Reflowing a long line until it wraps results in the cooked read
prompt to be misaligned vertically.
* `SCREEN_INFORMATION::s_RemoveScreenBuffer` should trigger a
buffer switch similar to `SetConsoleActiveScreenBuffer`.
* Translation of `COMMON_LVB_GRID_HORIZONTAL` to `SGR 53` was dropped
and may be reintroduced alongside `UNDERSCORE` = `SGR 4`.
* Move the `OSC 0 ; P t BEL` sequence to `WriteWindowTitle`
and swap the `BEL` with the `ST` (`ESC \`).
* PowerShell on Windows 10 ships with PSReadLine 2.0.0-beta2
which emits SGR 37/40 instead of 39/49. This results in black
spaces when typing and there's no good way to fix that.
* A test is missing that ensures that `FillConsoleOutputCharacterW`
results in a `CSI n J` during the PowerShell shim.
* A test is missing that ensures that `PtySignal::ClearBuffer`
does not result in any VT being generated.
Closes#262Closes#1173Closes#3016Closes#4129Closes#5228Closes#8698Closes#12336Closes#15014Closes#15888Closes#16461Closes#16911Closes#17151Closes#17313
This pull request removes the following vendored open source, in favor
of getting it from vcpkg:
- CLI11 2.4
- jsoncpp 1.9
- fmt 7.1.3
- gsl 3.1 (not vendored, but submoduled--arguably worse!)
Now that Visual Studio 2022 includes a built-in workload for vcpkg, the
onboarding process is much smoother. Terminal should only require the
vcpkg workload.
I've added some build rules that detect vcpkg via VS and via the user's
environment before falling back to a location in the source tree. The CI
pipeline will fall back to installing and bootstrapping vcpkg in
dep/vcpkg if necessary.
Some OSS has not been (and will not be) migrated:
- wyhash: ours is included directly in til/hash
- pcg_random: we have a stripped down copy compared to vcpkg
- stb_rect: vcpkg only ships *all of STB*; ours is a stripped down copy
- chromium numerics: vcpkg does not ship Chromium, especially not this
tiny fraction of Chromium
- dynamic_bitset and libpopcnt: removing in #17510
- interval_tree: no vcpkg equivalent
To support the needs of the inbox Windows build, I've split up our vcpkg
manifest into dependencies for all projects and dependencies just for
Terminal. To support this, we now offer a `terminal` feature. The vcpkg
rules in `common.build.pre.props` are set up to turn it on, whereas the
build rules we eventually write for the OS will not be.
Most of the work is concentrated in `common.build.pre.props`.
This is practically a from scratch rewrite of AtlasEngine.
The initial approach used a very classic monospace text renderer, where
the viewport is subdivided into cells and each cell is assigned one
glyph texture, just like how real terminals used to work.
While we knew that it would have problems with overly large glyphs,
like those found in less often used languages, we didn't expect the
absolutely massive number of fonts that this approach would break.
For one, the assumption that monospace fonts are actually mostly
monospace has turned out to be a complete lie and we can't force users
to use better designed fonts. But more importantly, we can't just
design an entire Unicode fallback font collection from scratch where
every major glyph is monospace either. This is especially problematic
for vertical overhangs which are extremely difficult to handle in a
way that outperforms the much simpler alternative approach:
Just implementing a bog-standard, modern, quad-based text renderer.
Such an approach is both, less code and runs faster due to a less
complex CPU-side. The text shaping engine (in our case DirectWrite)
has to resolve text into glyph indices anyways, so using them directly
for text rendering allows reduces the effort of turning it back into
text ranges and hashing those. It's memory overhead is also reduced,
because we can now break up long ligatures into their individual glyphs.
Especially on AMD APUs I found this approach to run much faster.
A list of issues I think are either obsolete (and could be closed)
or resolved with this PR in combination with #14255:
Closes#6864Closes#6974Closes#8993Closes#9940Closes#10128Closes#12537Closes#13064Closes#13527Closes#13662Closes#13700Closes#13989Closes#14022Closes#14057Closes#14094Closes#14098Closes#14117Closes#14533Closes#14877
## PR Checklist
* Enabling software rendering enables D2D mode ✅
* Both D2D and D3D:
* Background appears correctly ✅✅
* Text appears correctly
* Cascadia Code Regular ✅✅
* Cascadia Code Bold ✅✅
* Cascadia Code Italic ✅✅
* Cascadia Code block chars leave (almost) no gaps ✅✅
* Terminus TTF at 13.5pt leaves no gaps between block chars ✅✅
* ``"`u{e0b2}`u{e0b0}"`` in Fira Code Nerd Font forms a square ✅✅
* Cursor appears correctly
* Legacy small/medium/large ✅✅
* Vertical bar ✅✅
* Underscore ✅✅
* Empty box ✅✅
* Full box ✅✅
* Double underscore ✅✅
* Changing the cursor color works ✅✅
* Selection appears correctly ✅✅
* Scrolling in various manners always renders correctly ✅✅
* Changing the text antialising mode works ✅✅
* Semi-transparent backgrounds work ✅✅
* Scroll-zooming the font size works ✅✅
* Double-size characters work ✅✅
* Resizing while text is printing works ✅✅
* DWM `+Heatmap_ShowDirtyRegions` shows that only the cursor
region is dirty when it's blinking ✅✅
* D2D
* Margins are filled with background color ❌
They're filled with the neighboring's cell background color for
convenience, as D2D doesn't support `D3D11_TEXTURE_ADDRESS_BORDER`
* D3D
* Margins are filled with background color ✅
* Soft fonts work ✅
* Custom shaders enable continous redraw if time constant is used ✅
* Retro shader appears correctly ✅
* Resizing while a custom shader is running works ✅
Arm64EC does not support AVX and the usage of it in EC compilation is now an error with the LKG14 compiler update. The fix is to conditionalize using AVX for non-EC compilation.
Related work items: MSFT-42045281
Retrieved from https://microsoft.visualstudio.com os.2020 OS official/rs_we_adept_e4d2 31ca1e08e001988b95ff29a5e098441cae0363bd
This replaces ~70k LOC (parts of boost, 1/4th of the code in this project)
with ~700 LOC (`small_vector.h`). By replacing boost, we simplify future
maintenance and improve compile times.
## Validation Steps Performed
* New and existing unit tests are ok ✅
* Various common VT applications run fine in debug mode OpenConsole ✅
This pull request adds the JSON schema for `cgmanifest.json`.
## FAQ
### Why?
A JSON schema helps you to ensure that your `cgmanifest.json` file is valid.
JSON schema validation is a build-in feature in most modern IDEs like Visual Studio and Visual Studio Code.
Most modern IDEs also provide code-completion for JSON schemas.
### How can I validate my `cgmanifest.json` file?
Most modern IDEs like Visual Studio and Visual Studio Code have a built-in feature to validate JSON files.
You can also use [this small script](https://github.com/JamieMagee/verify-cgmanifest) to validate your `cgmanifest.json` file.
### Why does it suggest camel case for the properties?
Component Detection is able to read camel case and pascal case properties.
However, the JSON schema doesn't have a case-insensitive mode.
We therefore suggest camel case as it's the most common format for JSON.
### Why is the diff so large?
To deserialize the `cgmanifest.json` file, we use [`JSON.parse()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/parse).
However, to serialize the JSON again we use [`prettier`](https://prettier.io/).
We found that, in general, it gave smaller diffs than the default [`JSON.stringify()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/stringify) function.
wyhash was chosen based on the results found in `smhasher`, were it proved
itself as an algorithm with little flaws and fairly high output quality.
While I have a personal preference for xxhash (XXH3 specifically), wyhash is a
better fit for this project as its source code is multiple magnitudes smaller,
simplifying the review and integration into the header-only `hash.h` file.
For use with hashmaps the hash quality doesn't actually matter much for
optimal performance and instead the binary size usually matters more.
But even in that scenario wyhash is fairly close to FNV1a (aka "FNV64").
The result is that this new hash algorithm will only have little impact on
hashmap performance if used over the standard FNV1a as used in the STL,
while simultaneously offering a vastly better hash quality.
This partially solves #13124.
## Validation Steps Performed
* Added test cases ✅
This implements command line matching for `CascadiaSettings::GetProfileForArgs`.
The command lines for all user profiles are resolved to absolute file paths,
argument quotes are standardized ("canonicalized") and the results are cached.
When `GetProfileForArgs` is called with a Commandline() value, we "canonicalize"
the argument as well and find the profile that is the longest prefix.
If none could be found the default profile is returned.
## PR Checklist
* [x] Closes#9458
* [x] Closes#10952
* [x] I work here
* [ ] Tests added/passed
## Validation Steps Performed
* Open a `cmd.exe` tab in the store-version of WT
* Run `start cmd`
--> A tab with the `cmd.exe` profile opens
* Run `start pwsh.exe`
--> A tab with the PowerShell 7 profile opens
* Run PowerShell 7 from the start menu
--> A tab with the PowerShell 7 profile opens
* Create a symlink for PowerShell 7 and launch `pwsh.exe` from there
--> A tab with the PowerShell 7 profile opens
{fmt} 7.1.3 includes a number of changes, which I will summarize here:
* Switched to Dragonbox for float formatting (quoted, "20-30x faster"
than ostringstream)
* Significantly improves `FMT_COMPILE` (compile-time format string)
In my #8489 we want to use boost's small_vector type, but that PR is
kinda messy by adding boost and also making a meaningful change. So
here I'm splitting out the boost addition to its own PR so that one can
be more focused on the allocation improvement and consumption of boost.
This pull request is the initial implementation of hyperlink auto
detection
Overall design:
- Upon startup, TerminalCore gives the TextBuffer some patterns it
should know about
- Whenever something in the viewport changes (i.e. text
output/scrolling), TerminalControl tells TerminalCore (through a
throttled function for performance) to retrieve the visible pattern
locations from the TextBuffer
- When the renderer encounters a region that is associated with a
pattern, it paints that region differently
References #5001Closes#574
{fmt} 7.0.1 improves binary size, compile-time format string handling,
compile time improvements and named arguments.
In a test Windows build, it shrank our binary by ~14kb.
Closes#6905.
## PR Checklist
* [x] Closes#6905
* [x] CLA
This change is necessary as the dep/ folder is not synced into the
Windows source tree.
I've also added a build rule producing a lib for {fmt}.
This will be required for our next OS ingestion.
Leonard Hecker