Escape/transform U+001B when it appears in the Windows console

[rprichard]

Currently, if a console cell contains U+001B, winpty copies it through to the terminal. This behavior wasn't ever intentional, and now that I've noticed it, I think it should be changed, probably by replacing U+001B with another character like a space, a question mark, or perhaps U+FFFD (�). winpty generally tries to keep the terminal looking like the console, and the console does not interpret escape sequences.

This behavior is arguably a feature. For example, this currently works, assuming default colors and a wide enough mintty terminal:

$ WINPTY_SHOW_CONSOLE=1 console c:/Python27/python
Python 2.7.10 (default, May 23 2015, 09:40:32) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> print "\x1b[9mCROSSED\x1b[0m"
CROSSED
>>>

In mintty, the text is struck through, like CROSSED.

In the winpty's underlying console, this is what appears:

Python 2.7.10 (default, May 23 2015, 09:40:32) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> print "\x1b[9mCROSSED\x1b[0m"
�[9mCROSSED�[0m
>>>

The ability breaks easily, though. If the escape sequence wraps around a line, winpty will typically insert its own control sequences (or at least CR and LF) into the console's escape sequence. The ability also interferes with winpty's color handling and cursor positioning.

If a Windows console program's escape sequences are interpreted, they should be interpreted without writing them to the console screen buffer.

[rprichard]

See also: #43 and https://youtrack.jetbrains.com/issue/IDEA-155071

[rprichard]

I noticed today a few instances where '\x1b' seemed to be translated to something other than '?'.

• On Windows 7, in a console, run the Utf16Echo test program as Utf16Echo 1B. With CP437, it produces U+2190 (LEFTWARDS ARROW). With CP1252, it instead produces U+001B, and with CP65001, it produces U+0000. The same behavior can be observed running Python 2.7 in a Cygwin mintty terminal via winpty:

  $ winpty -Xallow-non-tty cmd /c chcp 1252 '&&' c:/python27/python -c 'print "7\x1b8"' | hexdump -C
  00000000  1b 5b 30 6d 41 63 74 69  76 65 20 63 6f 64 65 20  |.[0mActive code |
  00000010  70 61 67 65 3a 20 31 32  35 32 1b 5b 30 4b 1b 5b  |page: 1252.[0K.[|
  00000020  3f 32 35 6c 0d 0a 1b 5b  30 4b 1b 5b 3f 32 35 68  |?25l...[0K.[?25h|
  00000030  37 3f 38 1b 5b 30 4b 1b  5b 3f 32 35 6c 0d 0a 1b  |7?8.[0K.[?25l...|
  00000040  5b 30 4b 1b 5b 3f 32 35  68                       |[0K.[?25h|
  00000049
  

• It looks like PHP 7.1.3 changes the code page to 65001 when it runs. (PHP restores the original code page on exit, but not if you kill it with Ctrl-C. Suppose you run php with no arguments, then realize there's no PHP REPL...)

  winpty -Xallow-non-tty c:/php-7.1.3/php.exe -r 'print "7\x1b8";' | hexdump -C
  00000000  1b 5b 30 6d 1b 5b 30 4b  37 00 38 1b 5b 30 4b     |.[0m.[0K7.8.[0K|
  0000000f
  

In VSCode, the NUL character occupies 0 columns.

It might make sense to also translate U+0000 to '?'. (or maybe space?)

IIRC, I didn't originally translate U+001B to U+2190, because the left arrow codepoint's width is ambiguous, and it might have displayed as 2 columns in some CJK fonts on Windows.