os: Environ doc comment does not describe keys containing "="

[bcmills]

In investigating #49882, I noticed some other suspicious environment variables from os.Environ(). I'll write a regression test demonstrating the bug and then update here.

[rsc]

The "=FOO" is a real environment variable on Windows and it probably needs to be preserved in os.Environ, even if it is incompatible with os.Setenv. Setenv is a helper but not necessarily capable of everything.

[bcmills]

At the very least, we need to update the os.Environ docs, since they currently promise the form “key=value” and do not mention the possibility of either an empty key or one containing "=".

Moreover, it is common to use os.Environ() as the basis for the Env field in an exec.Cmd, which also does not document any specific behavior for keys beginning with =. If these variables can't be set, can they also not be transferred to subprocesses? (Or are they ignored, or can they be set only for subprocesses? If parts of Cmd.Env may be silently ignored, that also seems like a serious problem )

(@bufflig, @alexbrainman: some clarification on the standard Windows semantics might be helpful here.)

[gopherbot]

Change https://golang.org/cl/367850 mentions this issue: os: test that LookupEnv reports all keys found in Environ

[bcmills]

Empirically, it appears that if we parse through to the second =, both LookupEnv and Setenv can handle the resulting key.

It is not obvious to me how many = symbols the caller is expected to parse through in general. (For example: should a correct consumer include arbitrarily many leading = symbols, or just one?)

[bufflig]

Microsofts own docs (like https://docs.microsoft.com/en-us/windows/win32/api/processenv/nf-processenv-getenvironmentstrings) claims that environment names can not contain '='. They do not specifically mention empty environment variable names though, so I would personally interpret =C:=C:\golang as having the key '' and the value 'C:=C:\golang' - although that is probably not what was intended...

As the windows API for creating a (sub)process takes a null terminated block of null terminated strings 'name=value\0' with possibly just the convention of "environment variable names can not contain '='", it's possible you can technically set whatever you want, completely ignoring the key/value semantics. I guess we just have to behave as closely as possible to GetEnvironmentStrings, GetEnvironmentVariable and SetEnvironmentVariable. I can write some C programs to check how they behave in these situations.

[bcmills]

They do not specifically mention empty environment variable names though, so I would personally interpret =C:=C:\golang as having the key '' and the value 'C:=C:\golang' - although that is probably not what was intended...

Empirically the key in that example is =C:. (I tried both approaches in CL 367850, and the “include the leading = in the key” approach is what matched the observed behavior of LookupEnv.)

Another such key noticed during testing is =ExitCode.

I guess we just have to behave as closely as possible to GetEnvironmentStrings, GetEnvironmentVariable and SetEnvironmentVariable. I can write some C programs to check how they behave in these situations.

SGTM.

[bufflig]

Ok, it seems to behave like this in the barest MSVC program I could muster (completely empirically, I can find no docs describing this):

Environment variables beginning with exactly one '=' are allowed, you can set them and get them with SetEnvironmentVariable and GetEnvironmentVariable. No more than one single '=' in the beginning of a variable name is allowed. It's possible the interpretation should be that the first character can be whatever, as a key can not have zero characters.

When i.e. cmd.exe prints environments, these variables are not printed, i.e. they are "hidden", but they are in the environment block returned by GetEnvironmentStrings and can be retrieved by calling GetEnvironmentVariable in subprocesses (that inherit the environment).

You can add something that in the environment block looks like "===FOO=bar", either by using your own environment block when calling CreateProcess or by doing SetEnvironmentVariable("=","=FOO=bar"). But as expected, that can only be retrieved in a subprocess as GetEnvironmentVariable("=",...).

As a fun quirk/bug, you can actually do SetEnvironmentVariable("=FOO","bar=baz"); and then GetEnvironmentVariable("=FOO=bar",...) and get "baz", but only if you do it in the same process, a subprocess can only retrieve "=FOO" if it inherited the environment. I have no idea what kind of caching makes that "work" when you do it in the same process... I don't think we need to implement that.

I think the reasonable and compatible solution is to simply allow one prefix '=' in a key, and document that that is possible in Windows.

[alexbrainman]

I guess we just have to behave as closely as possible to GetEnvironmentStrings, GetEnvironmentVariable and SetEnvironmentVariable

I agree with @bufflig sentiment here. And so far I don't see any problems with our current code.

If anyone really interested to see where environment variables, like =C:, come from, see Undocumented Dynamic variables section in

https://ss64.com/nt/syntax-variables.html

and in particular

https://devblogs.microsoft.com/oldnewthing/20100506-00/?p=14133

Alex

[gopherbot]

Change https://go.dev/cl/401339 mentions this issue: os/exec: preserve original order of entries in dedupEnv