missing termios.h functionality

[mash-graz]

while trying to port a simple command line application to WASI and the console crate in particular, i stumbled over some unpleasant limitations concerning unbuffered keyboard input and suppressed echo output for passwd entries as required by many CLI applications.

i perfectly understand the motivations, why this features were not needed/available in CloudABI, and i respect the reservations to pick up a POSIX interface, which is full of unnecessary historic cruft and furthermore isn't supported on the windows platform, nevertheless there is not much alternative available except inventing the wheel once again. but without any intermediating system interface, we are simply not able to work around this grave limitation.

in practice we would most likely only need a small part of the features provided by termios.h resp. tcgetattr/tcsetattr/tcflush, but it's perhaps better for the practical portability of existing *nix code to just reuse this old fashioned POSIX interface the way it is. this should work on nearly any platform except windows, but even on this platform we could implement the needed features via SetConsoleMode etc. without much security worries.

the most essential feature missing are IMHO:

• unbuffered single key input
• suppressed echo for password entries

most other terminal control is anyway available via virtual terminal escape sequences (...although a more suitable forwarding of environment variables by the runtime implementations -- especially: TERM, TZ and the locale related ones -- would be also very helpful for this purpose resp. better CLI application support).

there are already similar closed feature requests available (eg. #42) and some brave developers are already looking for rather unorthodox workarounds (see: wasmerio/wasmer-js#141), nevertheless i really would like to get an idea, what we should expected as the desired solution for this issue in the long run.

[sunfishcode]

Thanks for reporting this! The current practice of assuming programs can use ANSI-style escape codes is actually problematic from an ecosystem perspective because the functionality differs between platforms. It'd be great to work together to find solutions here.

• unbuffered single key input
• suppressed echo for password entries

These seem like useful features

For TZ and LC_*/LANG/LANGUAGE

[dumblob]

Feel free to look how such highly cross-platform functionality is practically implemented in neovim forks which uses LeoNerd's libtermkey and libvterm for that.

[mash-graz]

Thanks for reporting this! The current practice of assuming programs can use ANSI-style escape codes is actually problematic from an ecosystem perspective because the functionality differs between platforms. It'd be great to work together to find solutions here.

thanks @sunfishcode for this warm words of welcome!

when i stumbled over this issue, i thought at first, i could prepare a sketch of a solution for this problem, but after spending a few days of studding the requirements, i came to the conclusion, that it is in fact far beyond my actual capabilities. nevertheless it's perhaps an interesting case, to debate the actual limitations of WASI and look for desirable solutions to workaround the given situation in an exemplary manner.

in one or the other way it seems to be necessary to extend the actual state of WASI to realize a solution for this issue. and that's indeed a serious endeavor, which shouldn't be handled in an unreflected ad hoc manner. unfortunately it's also affecting many different components -- i.e.: the WASI core interfaces/specification, WASI libc suport and the actual used runtimes --, so it's rather hard to even play around with experimental drafts of improvement code.

it may be also seen as an example, where Optional Imports resp. modularization could make a lot of sense.
e.g. for browser based runtime utilization or minimalist embedded IoT scenarios it's simply not necessary or useful to provide this kind of features and make everything bloated. but then again it's nearly indispensable for satisfying CLI application support.

platform specific handling

solving this particular requirement is a very tricky challenge, because it's by no means trivial to find a solution, which satisfies traditional POSIX termios.h utilizing system just the same as the windows platform by one single mechanism. the fundamental differences on both sides are very hard to overcome. in nearly all existing cross platform code this hindrance is solved by conditional compilation of different code segments. but that's a solution, which isn't compatible with WASIs general concept. we therefore have to chose between two other options:

• a. provide a new platform independent interface, which hides all the difference of the actual used system

or

• b. utilize a mechanism, like the one described in the Optional Imports design specification, to look for available terminal handling modules supported by the application, and realize the necessary execution branching at runtime.

for good reasons some of us may prefer a.) and there is already a multitude of intermediate terminal handling libraries and language runtimes available, which realize this approach. unfortunately many of them share the the same unpleasant characteristics: they all use different functions and control concepts in their wrappers, usually support only a small part of the actual system capabilities and tend to be very constricting and error prone in practice.

although i would always use one of the available mature intermediate terminal handling libraries for complexity reduction in case of practical user level development, because the native system interfaces are often hard to utilize in an economic manner, i still prefer variant b.), because it still preserves full access to all system specific capabilities in principle and makes it much easier to port existing code resp. replace the intermediate layer by some other library/crate whenever you like.

but it's indeed a puzzling question, which of this two approaches makes more sense? the pros and cons of both alternatives have be respected. but at the end it shouldn't make a big difference, which variant we choose, as long as the solution is realized in a modular fashion, which can be further improved over time.

general reservations concerning terminal feature utilization

The current practice of assuming programs can use ANSI-style escape codes is actually problematic from an ecosystem perspective because the functionality differs between platforms.

although i agree with with all your efforts to reduce the WASI core interface to the absolute necessary minimum, keep it secure and platform agnostic, i wouldn't see virtual terminal utilization as an hostile enemy and avoidable source of incompatibility. quite the opposite! in fact it provides a set of features, that would otherwise need system specific control interfaces and vendor specific obscurities. e.g. utilizing CSI-controll we don't need additional system specific functions to query the terminal size and cursor position, because this kind of information can be received in a rather system independent manner by just utilizing common in-band terminal communication whenever VT handling is enabled (e.g. by ENABLE_VIRTUAL_TERMINAL_PROCESSING). for a long time this worked only reliable on *nix machines, but in the meanwhile even windows boxes obtained remarkable improvements regarding this capabilities (see e.g.: https://devblogs.microsoft.com/commandline/).

it's also slightly misleading, if we associate more advanced terminal handling only with unnecessary colored output, emoji support and other fancy cosmetic crap. just think about the usefulness of readline command completion resp. history access. in this case you definitely don't want want to be forced to always press an additionally ↩️ key after every every ⬆️ keypress, just to recall the last command e.g. in a minimalist pure text mode debugger. and that's more the kind of usage, which i'm actually looking for. it's also a very simple example to illustrate, what we can't realize right now with WASI. if toggling un/buffered terminal mode and echo suppression would be possible via common escape sequences as well, i would immediately utilize this workaround and avoid to bother anybody with my desires.

conclusion

It'd be great to work together to find solutions here.

i'm sure, there are much more important tasks waiting to be solved by the WASI core developers, but perhaps this particular feature request could also serve as a demonstration, how this kind of improvements can be solved by a modular approach. such an comprehensible example would be very instructive for many similar cases. but that's perhaps just an utterly utopian fantasy.

i hope, you can excuse my lengthy elaborations and grasp at least some inspirations out of all this confuse rubbish.

[dumblob]

@mash-graz this is a great writeup, thanks!

I would emphasize the following from your post as it leads me to the question "What everything could be moved from WASI to the potential LanguageTerminal Server Protocol?":

although i agree with with all your efforts to reduce the WASI core interface to the absolute necessary minimum, keep it secure and platform agnostic, i wouldn't see virtual terminal utilization as an hostile enemy and avoidable source of incompatibility. quite the opposite! in fact it provides a set of features, that would otherwise need system specific control interfaces and vendor specific obscurities. e.g. utilizing CSI-controll we don't need additional system specific functions to query the terminal size and cursor position, because this kind of information can be received in a rather system independent manner by just utilizing common in-band terminal communication whenever VT handling is enabled (e.g. by ENABLE_VIRTUAL_TERMINAL_PROCESSING). for a long time this worked only reliable on *nix machines, but in the meanwhile even windows boxes obtained remarkable improvements regarding this capabilities (see e.g.: https://devblogs.microsoft.com/commandline/).

it's also slightly misleading, if we associate more advanced terminal handling only with unnecessary colored output, emoji support and other fancy cosmetic crap. just think about the usefulness of readline command completion resp. history access. in this case you definitely don't want want to be forced to always press an additionally  key after every every  keypress, just to recall the last command e.g. in a minimalist pure text mode debugger. and that's more the kind of usage, which i'm actually looking for. it's also a very simple example to illustrate, what we can't realize right now with WASI. if toggling un/buffered terminal mode and echo suppression would be possible via common escape sequences as well, i would immediately utilize this workaround and avoid to bother anybody with my desires.

[mash-graz]

@dumblob
thanks for your feedback!

and yes, this alternative would be also possible!

in fact it's a very old problem, how terminal communication can be serialized in a manner, which doesn't need any out of band control channels or direct API access.

telnet (RFC854) is such an example. it's in fact a transport layer agnostic protocol and could be used in principle also over very simple stdio communication. it's more oriented towards line buffered communication, because of it's performance benefits, but nevertheless it also provides commands to utilizes single character operation. it also comes with clear specified options for echo handling (RFC857), terminal-type identification (RFC1091), environment variable forwarding (RFC1572) etc.

so we could indeed use this protocol to tunnel al the needed features over the already existing WASI stdio infrastructure in a platform independent manner and just translate the control commands within the runtime on the host side to system specific API calls. but for our applications we would still have to provide some en/decoding facilities resp. interface again, to make this capabilities accessible. therefore i doubt, that all this de/serialization efforts will bring about any advantage in our case.

communication with the outside world isn't the crux of our problem and we should better avoid the performance penalties of unnecessary serialization overheap. we only have to look for a minimalist and efficient interface, which still satisfies the specific needs of the divergent native POSIX and windows APIs for the desired purposes.

[dumblob]

My point is, that we could leverage such "inner protocols" for problems which won't have any security implications and thus don't need to be part of the capabilities API. Therefore I'm afraid we can not "just reuse" an existing protocol as they're not designed with security in mind. But for some functionality it might make sense.

Security it pretty complicated thing - imagine that e.g. turning on colors could make an existing system vulnerable to phishing etc. by using similar looking colors to the default black&white etc. - but that's exactly the discussion we need to have first and decide up to which degree the capabilities should touch on such "inner protocols".

[mash-graz]

well -- security is always a compromise between useful limitation and preserving the actual capabilities resp. freedom of computing.

in this particular case i'm definitely convinced, that it doesn't make much sense to combat the dangers by any overly restrictive WASM design. in this respect it will hardly become better than the POSIX design, as long as we want to preserve a satisfaying base for more demanding command line applications -- e.g. editors.

the POSIX termio.h interface was already an specification attempt, to make the necessary calls more save and platform agnostic resp. portable than the underlying ioctl_tty calls.

as a long time linux user i'm well aware of all this [less] funny tricks to abuse terminal control sequences etc. nevertheless i respect and share the opinion of most security experts, that this risk can only be effectively solved by improving the utilized virtual terminal applications. that's the sole relevant place, where this kind of abuse can be prevented in a really consequent and reliable manner. but trying to handle it here in WASI is IMHO a hopeless ineffective endeavor.

and i really have to comment, that i want to use WASI for serious work, and not a just to play around with an utterly crippled toy, which isn't anymore usable e.g. for more powerful ncurses or readline based CLI software.

[sunfishcode]

@mash-graz

Yeah, there are several related topics. I've followed up on ANSI escape sequences and terminal input in the separate issues I've now opened for them so we can discuss those separately. Also, I apologize for accidentally posting an incomplete comment above. To finish my thought, I agree that locale and timezone info are useful, so I've now filed #167 to track that.

I agree that enabling unbuffered input and echo suppression are useful and pretty simple and we can probably do them without complication.

@dumblob

What everything could be moved from WASI to the potential LanguageTerminal Server Protocol?

Computing is full of "everything is a"s, isn't it ;-). It's great to bring ideas into the discussion; we also need help digesting them into actionable form.

[mash-graz]

I agree that enabling unbuffered input and echo suppression are useful and pretty simple and we can probably do them without complication.

this would be a really useful improvement!
it would indeed solve the requirement, which i originally couldn't work around.

and thanks for all your efforts to redirect the related topics in more useful individual tickets.

[bryzaguy]

This would be incredible! Especially if this led to an alternative to readline outside of WASI. I'm building a REPL and am shocked there aren't better, more widely supported solutions. I would leap at the chance to have people coding in my language in the browser.

[bryzaguy]

For context, a buffered solution doesn't allow you to do magical things like autocomplete as you type which feel really good.

[LinusU]

I ran into this whilst trying to compile my small CLI application to WASM/WASI. Does anyone know if there has been anymore progress on this outside of this issue?

[sunfishcode]

A while ago I started putting together a design related to this, which is documented here:

https://basic-text.sunfishcode.online/Terminal.html

It's not yet in practical shape, but it might be interesting as a starting point for discussion. Some thoughts;

• Stream-safe NFC and some of the Unicode rules might be too intrusive here.
• The approach of describing all supported escape sequences, instead of just letting escape sequences flow through to terminals to do whatever they do with them, makes the spec much longer and more complex than it otherwise would be, and restricts the set of features applications can use. On the other hand, it has a chance of ensuring that the features that are available should work in a portable way.

[Puellaquae]

It looks good. I'd to know whether it will become a proposal of wasi, and whether it's planned to provide some function to get info from terminal, such as terminal's size.

[TristanCacqueray]

It's a bit unfortunate WASI doesn't support unbuffered input that are necessary for interactive terminal games. Is this going to be ever be implemented in WASI, or should we look into WASIX which seems to handle TTY ?

[sunfishcode]

Building on the ideas in #162 and https://basic-text.sunfishcode.online/Terminal.html, and upcoming versions of wit-bindgen with handle support, I now have a draft PR here filling in the interfaces for terminal-input and terminal-output with an API that should support the most important terminal features, and it even gives some descriptions of escape codes which are expected to be supported.

A difference with the earlier approaches is that this doesn't require all implementations to parse and translate the bytestream. The escape codes described should have pretty consistent behavior across popular terminals. This has the downside of having nothing to prevent applications from having non-portable behavior, but this is for Preview 2, and we can re-evaluate that as WASI matures.

That said, implementations may still wish to parse the bytestream if they wish more control about which escape codes can be used or how they're interpreted, such as to control which information can be reported from the terminal back into the input of the application, to control how the output of an application can influence the output of other applications, or other things.

Feedback welcome!