input buffer doesn't clear out properly #397
-
I'm using clink with starship prompt, when I'm going through the history, the emojis are not cleared properly and still remain in the input buffer of the screen. Here's a video describing the issue: prompt.mp4I tried the same set of commands in other shells (fish) with the same starship configuration but couldn't reproduce the same issue, so it seems to be an issue with clink. Also a space should be automatically added next to the ">" symbol in the prompt. This space is displayed only for few of the history items, after that the space is not displayed for rest of the items. |
Beta Was this translation helpful? Give feedback.
Replies: 10 comments 8 replies
-
The video lets me see the problem. Can you also share your starship config, and your clink scripts, so that I can reproduce the issue and investigate? I think all the shells have problems with double width emoji like that, but the conditions where the problems are observable are different in different shells because they have different implementations for their input line display routines.
It looks like the terminal program is drawing the emoji using 2 characters. Clink and fish and bash and starship and etc have no way to know how the terminal will render various emoji and other glyphs. If the terminal program renders them using a different width than the current font calls for (e.g. double width instead of single width), then that ruins the shell's ability to accurately update the input line. Windows Terminal and mintty draw double width color emoji for some Unicode emoji codepoints, but other terminal programs use single width monochrome versions of the emoji, or just render them as a box. The outcome depends on many factors, but most of all on which terminal program is used. Bash and zsh and fish probably all use relative cursor positioning, which will often have a side effect of hiding the width mismatch until the input line wraps, or if the distance the cursor needs to move is large enough that it becomes more efficient to use absolute cursor positioning instead of relative cursor positioning. In the past, Clink's input line display used the Readline display routines from bash. But there were too many edge case problems (and especially when resizing the terminal width). I completely rewrote the input line display for Clink, and also made it always use escape codes for efficient absolute cursor positioning (bash and other *nix shells can't assume that the efficient absolute positioning escape codes are available, and they use less efficient alternatives). The rewrite has fixed the edge case problems, but it's had the side effect of making terminal rendering issues like double-width emoji or CJK ambiguous width codepoints more obvious -- because with absolute positioning the math mismatch always occurs, which also makes it much easier to reproduce and diagnose the terminal rendering problems and pinpoint that the cause is the terminal program itself. |
Beta Was this translation helpful? Give feedback.
-
Well, my starship configuration is very simple [character]
success_symbol = "🚀 [❯](bold green)"
[cmd_duration]
format = "took $duration ✔ "
load(io.popen('starship init cmd'):read("*a"))()
> clink info
version : 1.4.6.8b1dec |
Beta Was this translation helpful? Give feedback.
-
@chrisant996 Thanks for the quick & detailed explanation, going through the explanation, the rabbit hole seems to be quite deep here. Please do let me know if the problem is reproducible on the other end. |
Beta Was this translation helpful? Give feedback.
-
Please note that the previous video had the terminal window maximized, the behavior when the terminal is of normal dimensions (attached below) is observed to be slightly different. Note how the space after ">" is present only for the last two entries in the history (which I was talking about) and later it is removed as the input line gets updated. prompt2.mp4Not sure if this is a helpful detail, but I was unable to move the cursor to the right at 0:04 (probably this has something to do with the emoji width you were talking about?) |
Beta Was this translation helpful? Give feedback.
-
@VPraharsha03 Thank you for sharing all the extra info, that helped very much. The cause is indeed the issue I mentioned. It looks like the wezterm documentation mentions it in passing here.
More Explanation of The ProblemThe problem is that a console mode program historically has always expected that each character takes up one character cell on the screen. But some terminal programs have started using text drawing APIs that depart from that, and use two character cells to draw one character, but only for certain emojis (and sometimes even for non-emoji characters). A console mode program can't predict how a terminal program will draw each character or emoji, and console mode programs can't even reliably figure out what terminal program is hosting them in the first place. So this is mismatch between terminal program capabilities and console mode program expectations. Terminal programs will often say "the console mode program doesn't properly support emoji". But there is no standard or correct way to "support emoji". Programs have to hack things and hope that they are only used with a terminal program that supports color emoji, otherwise the opposite problem happens. Here is the official list of Unicode Emoji Sequences, and in your browser you'll notice that not all of the emoji are even rendered correctly -- some of them show up as a box or as a question mark inside a box. When a terminal program tries to display those emoji, then weird things happen. For example It's not entirely accurate to suggest that the terminal program is "doing everything correctly" and that the fault lies entirely with "programs that don't support emoji properly". There's no definition yet for how to support emoji properly; it's all a house of cards hack currently. It ends up working out very well ... as long as a program doesn't try to perform any cursor positioning operations. When those are used, then the mismatch between actual terminal rendering versus console program expectations of rendering collide and things go wrong. Console mode programs can very reasonably argue that, instead, terminal programs are not rendering color emoji properly. What Clink Might Be Able To Do About ItI can try adding a lookup table for "color emoji widths" that uses 2 as the width for color emojis from the official list of Unicode Emoji Sequences. Clink could use that table when it detects that it's running inside a terminal that supports color emoji. And I can try to find a way to detect when wezterm is used as the terminal. But that's a big hack, and it won't work properly for all emoji, depending on the specific OS version and terminal program version. The worst part is that terminal programs have glitches in rendering double-width color emoji (and even certain monochrome double-width emoji), and sometimes even if a console mode program uses "correct width math", the cursor positioning still ends up broken because of the problem illustrated above with the Furthermore, I've found that it's inconsistent what cursor column the OS will report versus what is drawn by a terminal program. Terminal programs often draw the screen differently than even the OS expects, which leads to even more problems. Short VersionThis isn't exactly a "Clink problem", and you can get similar rendering glitches to occur with bash and fish and zsh, it's just more convoluted how to get the glitches to happen. In fact, not only was I able to repro similar rendering glitches in bash, but I even got bash to crash! Simply by positioning the cursor and then backspacing a little bit after repeatedly inserting So you can see that this is not unique to Clink, and it's not as simple as "the shell program doesn't support emoji properly". I'll see what kind of hackery I can come up with that might get color emoji to work a little better in Clink, but it's inherently problematic at the present, for all shell programs (and for any program that needs to reposition the cursor after printing a color emoji, which of course all input editors need to do). |
Beta Was this translation helpful? Give feedback.
-
@VPraharsha03 I've implemented a hack which can assume emoji (as defined in the Unicode v15.0 standard) are rendered as double-width characters in Windows Terminal. The new The hack seems to work very well, at least for the cases you've shared. It can't ever be perfect because the Unicode standard constantly evolves, and different OS versions support different versions of the Unicode standard (and to varying degrees), and because of implementation details in different terminal programs. But overall, the hack seems to work pretty well. I'm looking for a way to detect when WezTerm is the current terminal program, so that |
Beta Was this translation helpful? Give feedback.
-
@VPraharsha03 Clink v1.4.7 has been published and includes the hack for color emojis. Let me know how it works for you. |
Beta Was this translation helpful? Give feedback.
-
@chrisant996 Thanks a lot for your time and efforts! Now everything works perfect 😄 on 1.4.7 |
Beta Was this translation helpful? Give feedback.
-
Yes, true. I could see the previous issues now with |
Beta Was this translation helpful? Give feedback.
-
The double-width emoji behavior repros even with HKCU\Console\ForceV2 = 0, so I'll make |
Beta Was this translation helpful? Give feedback.
@VPraharsha03 I've implemented a hack which can assume emoji (as defined in the Unicode v15.0 standard) are rendered as double-width characters in Windows Terminal. The new
terminal.color_emoji
setting defaults toauto
, which assumes double-width when using Windows Terminal, or otherwise single-width characters. It can be set toon
oroff
to force whether to assume the terminal supports color emoji.The hack seems to work very well, at least for the cases you've shared. It can't ever be perfect because the Unicode standard constantly evolves, and different OS versions support different versions of the Unicode standard (and to varying degrees), and because of implementation details in diff…