input buffer doesn't clear out properly

[VPraharsha03]

I'm using clink with starship prompt, when I'm going through the history, the emojis are not cleared properly and still remain in the input buffer of the screen.

Here's a video describing the issue:

prompt.mp4

I tried the same set of commands in other shells (fish) with the same starship configuration but couldn't reproduce the same issue, so it seems to be an issue with clink.

Also a space should be automatically added next to the ">" symbol in the prompt. This space is displayed only for few of the history items, after that the space is not displayed for rest of the items.

[chrisant996]

Here's a video describing the issue:

The video lets me see the problem.

Can you also share your starship config, and your clink scripts, so that I can reproduce the issue and investigate?

I think all the shells have problems with double width emoji like that, but the conditions where the problems are observable are different in different shells because they have different implementations for their input line display routines.

I tried the same set of commands in other shells (fish) with the same starship configuration but couldn't reproduce the same issue, so it seems to be an issue with clink.

• What terminal program is being used with Clink?
• What terminal program is being used with fish?
• How are you launching them?

Also a space should be automatically added next to the ">" symbol in the prompt. This space is displayed only for few of the history items, after that the space is not displayed for rest of the items.

It looks like the terminal program is drawing the emoji using 2 characters. Clink and fish and bash and starship and etc have no way to know how the terminal will render various emoji and other glyphs. If the terminal program renders them using a different width than the current font calls for (e.g. double width instead of single width), then that ruins the shell's ability to accurately update the input line.

Windows Terminal and mintty draw double width color emoji for some Unicode emoji codepoints, but other terminal programs use single width monochrome versions of the emoji, or just render them as a box. The outcome depends on many factors, but most of all on which terminal program is used.

Bash and zsh and fish probably all use relative cursor positioning, which will often have a side effect of hiding the width mismatch until the input line wraps, or if the distance the cursor needs to move is large enough that it becomes more efficient to use absolute cursor positioning instead of relative cursor positioning.

In the past, Clink's input line display used the Readline display routines from bash. But there were too many edge case problems (and especially when resizing the terminal width). I completely rewrote the input line display for Clink, and also made it always use escape codes for efficient absolute cursor positioning (bash and other *nix shells can't assume that the efficient absolute positioning escape codes are available, and they use less efficient alternatives). The rewrite has fixed the edge case problems, but it's had the side effect of making terminal rendering issues like double-width emoji or CJK ambiguous width codepoints more obvious -- because with absolute positioning the math mismatch always occurs, which also makes it much easier to reproduce and diagnose the terminal rendering problems and pinpoint that the cause is the terminal program itself.

[chrisant996]

Also:

• What version of Clink are you using?
• Can you share an excerpt of your clink_history file that contains lines with emoji that reproduce the problem?

[VPraharsha03]

@chrisant996

Can you also share your starship config, and your clink scripts, so that I can reproduce the issue and investigate?

Well, my starship configuration is very simple

[character]
success_symbol = "🚀 [❯](bold green)"

[cmd_duration]
format = "took $duration ✔ "

starship.lua

load(io.popen('starship init cmd'):read("*a"))()

What terminal program is being used with Clink?
What terminal program is being used with fish?
How are you launching them?

• The terminal used with clink in the video was wezterm.
• While testing it in fish (with starship prompt) I used the Windows Terminal. I also tested running cmd.exe (clink+starship) and going through the history items and observed the same problem that occurred in wezterm.
• I've not used any flags while launching them

What version of Clink are you using?

> clink info                                                                  
version          : 1.4.6.8b1dec

[chrisant996]

Can you share your clink_history file? You can email it to the address in my GitHub profile. Or share an excerpt from it that's able to reproduce the problem?

[VPraharsha03]

Ok here's an excerpt

|CTAG_1653199730_25161250_2200_0
E:\WezTerm-windows-20221119-145034-49b9839f\wezterm-gui.exe imgcat %USERPROFILE%\.config\wezterm\au-os-M.png
|	time=1672331181
E:\WezTerm-windows-20221119-145034-49b9839f\wezterm.exe imgcat %USERPROFILE%\.config\wezterm\au-os-M.png
|	time=1672332265
cchoco upgrade starship
|	time=1672332279
choco upgrade starship
|	time=1672332292
gsudo
|	time=1672381862
starship explain
|	time=1672389547
E:\WezTerm-windows-20221119-145034-49b9839f\wezterm.exe ls-fonts
|	time=1672389710
E:\WezTerm-windows-20221119-145034-49b9839f\wezterm.exe ls-fontss --list-system
|	time=1672389724
E:\WezTerm-windows-20221119-145034-49b9839f\wezterm.exe ls-fonts --list-system
|	time=1672390450
cd/d "E:\Self-Initiated Projects\"
|	time=1672391192
|:\WezTerm-windows-20221119-145034-49b9839f\wezterm.exe ls-fonts --text 🎈🎺🎉🎊
|	time=1672391301
E:\WezTerm-windows-20221119-145034-49b9839f\wezterm.exe ls-fonts --text 🎈🎺🎉🎊
|	time=1672391345
cls
|	time=1672391329
cd ~

[VPraharsha03]

@chrisant996 Thanks for the quick & detailed explanation, going through the explanation, the rabbit hole seems to be quite deep here. Please do let me know if the problem is reproducible on the other end.

[VPraharsha03]

Please note that the previous video had the terminal window maximized, the behavior when the terminal is of normal dimensions (attached below) is observed to be slightly different. Note how the space after ">" is present only for the last two entries in the history (which I was talking about) and later it is removed as the input line gets updated.

prompt2.mp4

Not sure if this is a helpful detail, but I was unable to move the cursor to the right at 0:04 (probably this has something to do with the emoji width you were talking about?)

[chrisant996]

@VPraharsha03 Thank you for sharing all the extra info, that helped very much.

The cause is indeed the issue I mentioned.

It looks like the wezterm documentation mentions it in passing here.

... For example, pasting some emoji into the zsh REPL confuses its input parser and results in broken emoji output. However, if you were to emit that same emoji from a script, wezterm would render it correctly. ...

More Explanation of The Problem

The problem is that a console mode program historically has always expected that each character takes up one character cell on the screen. But some terminal programs have started using text drawing APIs that depart from that, and use two character cells to draw one character, but only for certain emojis (and sometimes even for non-emoji characters). A console mode program can't predict how a terminal program will draw each character or emoji, and console mode programs can't even reliably figure out what terminal program is hosting them in the first place.

So this is mismatch between terminal program capabilities and console mode program expectations. Terminal programs will often say "the console mode program doesn't properly support emoji". But there is no standard or correct way to "support emoji". Programs have to hack things and hope that they are only used with a terminal program that supports color emoji, otherwise the opposite problem happens.

Here is the official list of Unicode Emoji Sequences, and in your browser you'll notice that not all of the emoji are even rendered correctly -- some of them show up as a box or as a question mark inside a box. When a terminal program tries to display those emoji, then weird things happen. For example (🫷..🫸) is copy/pasted from the official list, and should be a leftward pushing hand and a rightward pushing hand. But your browser will probably show it as two boxes. If you copy/paste it into a terminal program, then it will probably end up rendering as (□..□ ) which is especially crazy because the terminal ends up using 1 cell for the first unrecognized emoji, and 3 cells for the second unrecognized emoji! It's a good illustration of the underlying problems in how terminal programs (and the DirectDraw APIs for drawing color emoji) are trying to support color emoji.

It's not entirely accurate to suggest that the terminal program is "doing everything correctly" and that the fault lies entirely with "programs that don't support emoji properly". There's no definition yet for how to support emoji properly; it's all a house of cards hack currently.

It ends up working out very well ... as long as a program doesn't try to perform any cursor positioning operations. When those are used, then the mismatch between actual terminal rendering versus console program expectations of rendering collide and things go wrong. Console mode programs can very reasonably argue that, instead, terminal programs are not rendering color emoji properly.

What Clink Might Be Able To Do About It

I can try adding a lookup table for "color emoji widths" that uses 2 as the width for color emojis from the official list of Unicode Emoji Sequences. Clink could use that table when it detects that it's running inside a terminal that supports color emoji. And I can try to find a way to detect when wezterm is used as the terminal.

But that's a big hack, and it won't work properly for all emoji, depending on the specific OS version and terminal program version.

The worst part is that terminal programs have glitches in rendering double-width color emoji (and even certain monochrome double-width emoji), and sometimes even if a console mode program uses "correct width math", the cursor positioning still ends up broken because of the problem illustrated above with the (🫷..🫸) example.

Furthermore, I've found that it's inconsistent what cursor column the OS will report versus what is drawn by a terminal program. Terminal programs often draw the screen differently than even the OS expects, which leads to even more problems.

Short Version

This isn't exactly a "Clink problem", and you can get similar rendering glitches to occur with bash and fish and zsh, it's just more convoluted how to get the glitches to happen.

In fact, not only was I able to repro similar rendering glitches in bash, but I even got bash to crash! Simply by positioning the cursor and then backspacing a little bit after repeatedly inserting 🚀 [❯](bold green).

So you can see that this is not unique to Clink, and it's not as simple as "the shell program doesn't support emoji properly".

I'll see what kind of hackery I can come up with that might get color emoji to work a little better in Clink, but it's inherently problematic at the present, for all shell programs (and for any program that needs to reposition the cursor after printing a color emoji, which of course all input editors need to do).

[VPraharsha03]

Thanks for taking your time to explain the intricacies of the problem which I was unaware of (almost all the issues that were related to emojis/unicode with terminals till now have been quite complex). I do accept the above rationale. I'd to start somewhere and I guess I've posted it in the right place already :)

[chrisant996]

@VPraharsha03 I've implemented a hack which can assume emoji (as defined in the Unicode v15.0 standard) are rendered as double-width characters in Windows Terminal. The new terminal.color_emoji setting defaults to auto, which assumes double-width when using Windows Terminal, or otherwise single-width characters. It can be set to on or off to force whether to assume the terminal supports color emoji.

The hack seems to work very well, at least for the cases you've shared. It can't ever be perfect because the Unicode standard constantly evolves, and different OS versions support different versions of the Unicode standard (and to varying degrees), and because of implementation details in different terminal programs. But overall, the hack seems to work pretty well.

I'm looking for a way to detect when WezTerm is the current terminal program, so that auto can include it.
Update: Clink is now able to detect when the terminal host is WezTerm, and automatically defaults to assuming WezTerm supports color emoji.

[chrisant996]

@VPraharsha03 Clink v1.4.7 has been published and includes the hack for color emojis. Let me know how it works for you.

[VPraharsha03]

@chrisant996 Thanks a lot for your time and efforts! Now everything works perfect 😄 on 1.4.7

[VPraharsha03]

Tested on both Windows Terminal & wezterm

[VPraharsha03]

It can't ever be perfect because the Unicode standard constantly evolves, and different OS versions support different versions of the Unicode standard (and to varying degrees), and because of implementation details in different terminal programs.

Yes, true. I could see the previous issues now with conhost.exe and Hyper Terminal

[chrisant996]

By "conhost.exe", I think you mean the default legacy terminal in Windows. It does not show color emoji, so it does not have the issue.

Last I knew, hyper terminal also doesn't show color emoji.

[VPraharsha03]

To be more specific, I'm using OpenConsole (microsoft/terminal#8101) which clearly doesn't seem to have any kind of support for color emojis, but it's strange that I was able to repro the same issues.

Here's what I captured on OpenConsole (just for information)

OpenConsole.mp4

And Hyper does seem to have color emoji support.

[chrisant996]

To be more specific, I'm using OpenConsole (microsoft/terminal#8101) which clearly doesn't seem to have any kind of support for color emojis, but it's strange that I was able to repro the same issues.

On Win10 22H2, I'm able to reproduce the width issue even when not using Windows Terminal (i.e. just using a default legacy console window). I don't think that used to be the case in some older builds of Windows, but maybe I'm remembering inaccurately.

OpenConsole is conhost (see here). The main difference is that it's named "OpenConsole" when it's the updated one included in the Windows Terminal installation, and it's named "conhost" when it's the copy that's included as part of Windows itself.

The Windows version of the console host lags behind the Windows Terminal version, and some changes work their way back into the Windows version. So at some point in time, it was probably true that "conhost" didn't reproduce the double-width metrics for the Unicode codepoints that correspond to color emoji. But in Win10 22H2, at least, it does.

And that's the problem: there is no way for a program to ask the OS or console host how it's going to render individual Unicode codepoints. So programs must make guesses.

I'm reasonably comfortable making Clink by in sync with "current" builds of OS versions (e.g. Win10 22H2 and Win11 22H2), and requiring manual configuration of clink set terminal.color_emoji to select behavior that matches older OS versions when running on older OS versions. So, I'll adjust the terminal.color_emoji setting's auto mode to assume double-width characters (essentially making it the same as two for the moment, except that it can be adjusted in the future according to special cases that come up).

And Hyper does seem to have color emoji support.

It's because conhost (aka the built-in copy of OpenConsole) now always uses the DirectDraw metrics for color emoji. So, now all terminal host programs gain the double-width behavior, even if they're using the built-in conhost instead of their own console host. I'm pretty sure that wasn't always the case, as noted above.

[chrisant996]

The double-width emoji behavior repros even with HKCU\Console\ForceV2 = 0, so I'll make auto unconditionally behave the same as two (until someone reports a case where it yields unexpected behavior).