Windows compiler invocation opens multiple console windows

[Systemcluster]

This was previously reported under #16 and supposedly fixed in this commit and again in #472 (which I presume was included in the 0.2.10 release), but the issue still persists - upon the start of a compilation process (in particular, a cargo invocation), multiple console windows pop up for a split second.

I'm encountering this with the following environment:

• sccache version 0.2.11 installed through cargo (it also occured with 0.2.10 and previous versions)
• Windows 10 (x64) version 10.0.18362.0
• Running the cargo subcommands run, build or install, with the environment variable RUSTC_WRAPPER=sccache set and using the Rust msvc toolchain.

Please let me know if you need more information.

[tangmi]

@Systemcluster Are you using PowerShell? Do you mind giving your builds a shot through Command Prompt (ensuring that sccache is not already running)? I remember reading somewhere that PowerShell does not like CreateProcessW's DETACHED_PROCESS flag (edit: seems like it was an offhand comment on a StackOverflow post, but I'm not sure what to make of that)...

edit 2: We may be able to justify removing DETACHED_PROCESS since sccache owns the code for the spawned process (and we can just document run_server_process to not call AttachConsole(ATTACH_PARENT_PROCESS), which apparently is what DETACHED_PROCESS prevents.

[Systemcluster]

@tangmi I was using PowerShell, but I'm seeing the same behavior with CMD. Same with sccache 0.2.12-alpha.0 compiled with the changes from your branch as well.

[Systemcluster]

@tangmi I tried around a bit with some more start parameters, and after some experimentation, I decided to perform some logging with some thread::sleep calls to make it easier to pinpoint the source. As it turns out, the console popups are happening much later than the call to connect_or_start_server. (The initical sccache server start seems to create a very short lasting element in the taskbar, but no focus-stealing barrage of console windows as happening later.)

As far as I can tell (the sccache source is not the easiest to permeate) the popups appear inside run_input_output in util.rs. Which is very unfortunate, because at that point the process creation seems to be outsourced to the Rust libstd.

[gilescope]

Happens on a vanilla win7 cmd console also.

[Shadlock0133]

Happens also on windows 8 in msys2's mintty.

[gilescope]

And win10

_{Sent with GitHawk}

[galich]

it's a huge problem when running in Visual Studio Code + Rust Analyzer.

Rust Analyzer invokes cargo check frequently while editing code and it starts to show hundreds of these windows that close themselves in a moment.

i believe we can not afford DETACHED_PROCESS anymore:

CREATE_NO_WINDOW... is ignored if the application is... used with either CREATE_NEW_CONSOLE or DETACHED_PROCESS.

https://docs.microsoft.com/en-us/windows/win32/procthread/process-creation-flags

[captchaKing]

For those who whant to change this behavior without recompilation:
Search for C744242808060008 (on my binary it is address 0x2b32A8) and change it to C744242800060008

Don't forget to turn off sccache server beforehand (sccache.exe --stop-server).

If interested, bytes 08060008 correspond to CREATE_NO_WINDOW(0x8000000) | CREATE_UNICODE_ENVIRONMENT(0x400) | CREATE_NEW_PROCESS_GROUP(0x200) | DETACHED_PROCESS(0x8) considering endianness

[glandium]

My understanding of the problem is that the server being detached and not having a window, when itself spawns new processes, that may create a new window for the new process. I can imagine removing DETACHED_PROCESS might fix it, but I can also see it preventing the server itself from spawning properly in the first place.
It seems to me a better fix would be to use CREATE_NO_WINDOW when spawning processes from the server itself. Unfortunately, libstd doesn't have flags for that, so we'd need to use CreateProcess manually.
It also seems CREATE_NO_WINDOW should be removed from src/commands.rs, because it does nothing with DETACHED_PROCESS.

[Systemcluster]

but I can also see it preventing the server itself from spawning properly in the first place.

Can you elaborate on this? I've used my fork for a couple months now and I noticed no such issues.

[glandium]

Isn't the first sccache process blocked until the server ends if you don't use --start-server?

[glandium]

Ok, poking around, I've convinced myself that the flag that matters the most for my concerns is CREATE_NEW_PROCESS_GROUP, and that removing DETACHED_PROCESS seems fine.

[glandium]

This is fixed by the merge of #525, but I also just discovered a side effect while I was looking with process explorer, which is that there is now a conhost process attached, which there isn't with DETACHED_PROCESS. Presumably, that's what the things that were opening new windows end up attached to, which is why windows don't show up anymore.
In the end I think we'll have to go with what was hilighted in #514 (comment)
I'll keep this open until that's addressed, and will keep it on the radar for upcoming release.

[glandium]

Unfortunately, libstd doesn't have flags for that, so we'd need to use CreateProcess manually.

Actually, there is https://doc.rust-lang.org/std/os/windows/process/trait.CommandExt.html#tymethod.creation_flags

[glandium]

Interestingly, add cmd.creation_flags(CREATE_NO_WINDOW) makes the output from rustc entirely eaten (e.g. no warnings when there are warnings, no error messages in case of failure...)

[luser]

Actually, there is https://doc.rust-lang.org/std/os/windows/process/trait.CommandExt.html#tymethod.creation_flags

Humorously, that exists because I implemented it after having to do this the hard way when originally writing the Rust port of sccache. 🙂

[glandium]

At this point, I think we can't do better than the current (new) status, but I'll keep this open to still further investigate.

[mitchhentges]

I had a chat with glandium yesterday, and it sounds like this should be solvable today, though reading through this ticket it's currently a little bit vague what the correct solution is, and a little investigation is needed.

I can dig into this once I've perused through the remaining open PRs, but if someone else would like to snatch this up, then that would allow the next release to be pushed out faster - I'd like for the fix for this to be included as part of it.

[FooBarWidget]

I am on Windows Server 2022, sccache 21185d9, Rust msvc toolchain. I cannot reproduce the issue. Neither with cmd.exe, nor with PowerShell. I used sccache on a cargo build and verified that cargo wraps the rustc invocation with sccache.

Maybe this is because I start the server separately. I'll test later what happens if I let sccache start the server automatically.

[FooBarWidget]

I've also tested what happens if the let sccache start the server automatically. I still cannot reproduce the issue.

I've also researched the history of this issue, the relevant Windows APIs, and sccache's codebase as of commit 21185d9. I believe that this issue is now solved.

Here's how I arrived at this conclusion:

How does the Windows console work?

Windows differentiates between console apps and GUI apps. Windows knows whether an app is a console or a GUI app based on a marker in the executable that's set during linking.

What happens when one starts a console app:

1. If there's no console (e.g. the app was started from Windows Explorer) then Windows automatically allocates a console.
2. If there is already a console (e.g. the app was started from cmd.exe, PowerShell, Windows Terminal, etc.) then Windows attaches the app to the existing console.
3. If the app was started with DETACH_PROCESS or CREATE_NO_WINDOW, then Windows will neither create a new console, nor attach to the existing console. The app's won't have a console.

Okay let's say a console app (A) is started with DETACH_PROCESS. What happens if A starts another console app B, without any special CreateProcess() flags? Answer: B will be started in a new console!

But let's say console app A is started with CREATE_NO_WINDOW. If A starts another console app B, then B won't have a new console. This is not well-documented on MSDN, but I tested this today.

Note that if both DETACH_PROCESS and CREATE_NO_WINDOW are set, then Windows ignores CREATE_NO_WINDOW.

Why do I think the issue is now solved?

sccache appears to have two places that start console processes (and thus that can potentially make Windows create consoles):

1. In run_input_output(). Since c70dcf2, this function starts processes with the no_console() flag, which on Windows translates to CREATE_NO_WINDOW.
2. In run_server_process() which is for auto-starting the sccache server as a daemon. Right now, we start the server with CREATE_NO_WINDOW.

Since CREATE_NO_WINDOW's effect applies to the child process's children too, there shouldn't be any way for consoles to pop up. I certainly haven't been able to observe this bug.

I believe this issue can now be closed.

Other remarks

I don't think run_server_process() needs to perform unsafe calls to CreateProcessW() directly. As of Rust 1.16.0, it's now possible to pass Windows process creation flags directly to std::process::Command using std::os::windows::process::CommandExt::creation_flags.

[FooBarWidget]

I continued researching in order to further verify, or falsify, my hypothesis that the issue is now solved.

Artificially reproducing the problem

I can't reproduce the problem. But is that because the code is now fixed, or is that because there's something in my environment that prevents the problem from occurring? In order to answer this question, I tried to artificially induce the problem.

run_input_output() did not used to set any flags, while run_server_process() used to set DETACH_PROCESS (with CREATE_NO_WINDOW being ignored because of that). Thus, I expect that I will see the problem if do these:

1. Make run_input_output() not set any flags.
2. Make run_server_process() set DETACH_PROCESS instead of CREATE_NO_WINDOW.

After having made these changes, I indeed observed the problem: during an sccache invocation (with the server being auto-started) I observed lots of little console window popups.

Thus, I conclude that the use of CREATE_NO_WINDOW actually solves the problem, and that it's not just my own environment that's suppressing the problem.

Conclusion

The conclusion is still: the issue is fixed.

[mitchhentges]

Thanks for the great investigation comments, those are helpful 👍

I think that this ticket has shifted since its initial report:

1. Initially, multiple console windows were opened when sccache starts compilation processes
2. fix(windows): remove DETACHED_PROCESS flag when starting the server process #525 was merged, which made multi-console window no longer occur
3. Glandium raised this comment, where he mentioned that the issue was fixed, but now there's a bunch of extra conhost processes. I'm reading between the lines here and assuming that that impacts performance in a non-trivial way.
4. It sounds like the solution proposed here is ideal, but this later comment contradicts that saying that perhaps we can't do better than opening conhost processes.

So, in summary:

• An ideal solution would not open console windows, but also wouldn't start conhost processes in the background
• The solution for doing ^ is a bit murky, so I'll chat with glandium for clarification - this will probably happen on Monday, March 14th.

[mitchhentges]

The solution for doing ^ is a bit murky, so I'll chat with glandium for clarification - this will probably happen on Monday, March 14th.

Glandium has confirmed that he's comfortable deferring this ticket to a future release once we narrow down if there's still any possible improvement here.

[FooBarWidget]

I've researched the conhost issue.

What is conhost?

Conhost is a process that manages a console and its I/O. When you run cmd.exe and you see that black window — that's a console, backed by one conhost process. Each console is backed by a different conhost process.

Normally, a cmd.exe or PowerShell session is backed by one conhost process. When you run additional CLI processes from cmd.exe/PowerShell, they make use of the same console that cmd.exe/PowerShell also uses (they inherit the console), and so there is only one conhost process.

Conhost and DETACHED_PROCESS

When you spawn a console process B with DETACHED_PROCESS, that process won't have a console window, and thus it won't have a conhost.

But when B spawns a console process C — without any flags — then C will have a console. You can see a black window popping up. It will also have a corresponding conhost. This is consistent with my explanation in #514 (comment)

A
 |
 +-- B (spawned with DETACHED_PROCESS; no console; no conhost)
     |
     +-- C (no spawn flags; has a console; has a conhost)

Conhost and CREATE_NO_WINDOW

When you spawn a console process B with CREATE_NO_WINDOW, then that process will have a console. It's just hidden. Thus, B does have a conhost.

If B spawns a console process C with CREATE_NO_WINDOW, then C will also have a hidden console, and thus a corresponding conhost. Let's call this exhibit 1.

A
 |
 +-- B (spawned with CREATE_NO_WINDOW; has a hidden console; has a conhost)
     |
     +-- C (spawned with CREATE_NO_WINDOW; has a hidden console; has a conhost)

Above: exhibit 1 in action

But if B spawns a console process C with no extra flags, then C will inherit B's hidden console. And so C won't have a conhost.

A
 |
 +-- B (spawned with CREATE_NO_WINDOW; has a hidden console; has a conhost)
     |
     +-- C (no spawn flags; reuses B's hidden console; reuses B's conhost)

What does this mean for sccache?

The problem is that sccache hits exhibit 1.

1. An sccache invocation spawns its server with CREATE_NO_WINDOW. This results in a conhost.
2. The server spawns processes with CREATE_NO_WINDOW. This also results in a conhost per spawned process.

Proposed solution

Do both of these things:

1. Do spawn the sccache server with CREATE_NO_WINDOW.
2. From the server, don't spawn processes with CREATE_NO_WINDOW. Just don't use any flags. Concretely, this means that in run_input_output we should get rid of the no_console() call.

If we do these, then the sccache server will have a conhost, but there will be no further conhosts.

There is only one potential caveat, but it's a far-fetched corner case that I don't think anybody will run into:

• If the server was started manually, AND also specifically with the DETACH_PROCESS flag, then all console subprocesses that the server spawns will pop up console windows.

[mitchhentges]

Nice, good dig, thanks @FooBarWidget!

A couple things:

• I'm really happy with the solution of CREATE_NO_WINDOW, 👍
• The caveat of manual-server-start and DETACH_PROCESS is something that I'm OK with deferring to the future. If the edge case becomes an issue, we may be able to query the win32 API at server-start and log a warning if we've been invoked with DETACH_PROCESS.
• If my understanding is correct, then with the strong proposed solution of using CREATE_NO_WINDOW, we're going to have two conhosts, right? One for the currently running sccache command, and one for the sccache daemon and its subprocesses. I'm curious: is it viable to lean on DETACHED_PROCESS instead, so that we drop to one conhost? As you said, this is only viable if every descendant of the sccache daemon is created with DETACHED_PROCESS, but is that possible? I suppose that we're at risk of, say, cargo directly spawning a new subprocess (let's say process D to continue with your convention): in which case D will pop up in a new window, eh? If it were possible to use DETACHED_PROCESS for the daemon and all of its descendants, that could result in additional performance wins, perhaps.

[FooBarWidget]

If my understanding is correct, then with the strong proposed solution of using CREATE_NO_WINDOW, we're going to have two conhosts, right? One for the currently running sccache command, and one for the sccache daemon and its subprocesses.

Correct.

I'm curious: is it viable to lean on DETACHED_PROCESS instead, so that we drop to one conhost? As you said, this is only viable if every descendant of the sccache daemon is created with DETACHED_PROCESS, but is that possible? I suppose that we're at risk of, say, cargo directly spawning a new subprocess (let's say process D to continue with your convention): in which case D will pop up in a new window, eh? If it were possible to use DETACHED_PROCESS for the daemon and all of its descendants, that could result in additional performance wins, perhaps.

This is not viable. DETACHED_PROCESS only applies to the immediate process that we spawn, not to any subprocesses that it may spawn. If the sccache daemon spawns rustc with DETACH_PROCESS, and rustc spawns the linker, then the linker will cause a console window to pop up.

The only flag that extends to subprocesses is CREATE_NO_WINDOW.

[mitchhentges]

Assuming no other Windows shenanigans to influence transitive descendent processes, CREATE_NO_WINDOW sounds like the ideal solution. Ship it 🚢 my friend 👍