Error code ambiguity

[abrown]

Over in Wasmtime, the CLI binary rewrites certain exit codes from WASI's proc_exit in order to avoid issues on Windows; see here. This makes it difficult to write WASI tests (e.g., in wasi-testsuite) that reliably emit the same exit code across operating systems. I discovered this in bytecodealliance/wasmtime#5907 and will temporarily disable the tests on Windows.

I propose that we decide whether the exit code translation that Wasmtime currently does is something that all WASI engines should do. If so, then we can document that translation here and then the wasi-testsuite can either: a) create test runners that understand the exit code translation or b) write tests that avoid the ambiguity entirely.

cc: @sunfishcode, who has more context on this decision.

[codefromthecrypt]

I would note that error code ambiguity is only one part of behavioral differences for windows, and caution about WASI trying to specify too many things, especially if the motivation is the test suite and not an end user.

One alternative is similar to other parts in the test suite, tolerate multiple possible outcomes. For example, windows host behavior on abort (similarly some errors you can't get like trying to perform ops on a closed file.. these are different on windows).

In other words, rather than try to canonicalize an open box, could we move this to a range of acceptable answers in the test suite?

[abrown]

@codefromthecrypt, you might want to also add your vote to WebAssembly/wasi-cli#11. I think @sunfishcode is concluding that trying to specify these exit codes is too difficult; I'm fine with not specifying it as well.

[yamt]

for a testsuite, i guess it's simpler to use an embedder api to get the exit code.
(rather than trying to extract it from host exit code.)