ExecutionEngineException when hooking/detouring GetSysColor()

[FenPhoenix]

Description

I have an app that hooks/detours a handful of Windows functions as part of custom dark mode theming for WinForms. Those functions are:

• DrawThemeBackground()
• GetThemeColor()
• GetSysColor()

The behavior:

• In .NET Framework 4.8, the detoured versions of the functions all work fine in both x86 and x64 modes.
• In .NET 5, all of them work fine in x86 mode, but in x64 mode, the detoured GetSysColor() crashes with an ExecutionEngineException when called.
• In .NET 6 and 7, GetSysColor() produces the above crash in both x86 and x64 mode.
• DrawThemeBackground() and GetThemeColor() work fine in all abovementioned versions in both x86 and x64 modes.

The minimal repro uses MinHook.NET to demonstrate, but the same crash occurs in the same way with every hooking library I can find (Reloaded.Hooks, CoreHook, MinSharp). I even tried making a small C++ dll that uses MinHook (the original native version) and sets up the hooking entirely on the native side, with the .NET side simply calling a single "Start the hooking stuff" function and not actually p/invoking anything else through, but it still crashes in the same way.

Apologies if this should go in the WinForms repo; GetSysColor() might be considered WinForms-adjacent-ish, but it's possible that it's just a coincidence and some detours just crash and this is one of them, not sure.

Reproduction Steps

https://fenphoenix.com/getsyscolor/GetSysColorHookTest.zip

1. Open the solution in Visual Studio (we need to see the Trace.WriteLine() output in the console).
2. Set to x86 mode or x64 mode.
3. Run the app and click the "Enable DrawThemeBackground, GetThemeColor hooks" button.
4. Note at least one of each of the following lines in the console:
   "Now inside the body of hooked function GetThemeColor_Hooked"
   "Now inside the body of hooked function DrawThemeBackground_Hooked"
5. Click the "Enable GetSysColorHook" button.
6. The app crashes with an ExecutionEngineException.
7. In GetSysColorHookTest.csproj, change the TargetFramework to "net5.0-windows". Clean Solution and rebuild.
8. Set to x86 mode.
9. Repeat the above steps. Note that clicking the "Enable GetSysColorHook" button results in the message "Now inside the body of hooked function GetSysColor_Hooked" in the console and no crash.
10. Set to x64 mode.
11. Repeat the above steps and now note the ExecutionEngineException again.
12. In GetSysColorHookTest.csproj, change the TargetFramework to "net48". Clean Solution and rebuild.
13. Set to x86 mode.
14. Repeat the above steps. Notice no crash.
15. Set to x64 mode.
16. Repeat the above steps. Also no crash.

Expected behavior

The detoured GetSysColor() function runs and returns correctly with no crash.

Actual behavior

The detoured GetSysColor() function crashes with an ExecutionEngineException.

Clicking "Show call stack" brings up:

 	[Managed to Native Transition]	
 	System.Windows.Forms.dll!System.Windows.Forms.Application.ComponentManager.Interop.Mso.IMsoComponentManager.FPushMessageLoop(nuint dwComponentID, Interop.Mso.msoloop uReason, void* pvLoopData)	Unknown
 	System.Windows.Forms.dll!System.Windows.Forms.Application.ThreadContext.RunMessageLoopInner(Interop.Mso.msoloop reason, System.Windows.Forms.ApplicationContext context)	Unknown
 	System.Windows.Forms.dll!System.Windows.Forms.Application.ThreadContext.RunMessageLoop(Interop.Mso.msoloop reason, System.Windows.Forms.ApplicationContext context)	Unknown
>	GetSysColorHookTest.dll!GetSysColorHookTest.Program.Main() Line 19	C#

Clicking "Copy details" simply results in:

System.ExecutionEngineException
  HResult=0x80131506
  Source=<Cannot evaluate the exception source>
  StackTrace:
<Cannot evaluate the exception stack trace>

Clicking "View details" brings up a "Quick Watch" window with a large amount of information but I can't find a way to copy it out, or if it's even relevant.

Regression?

Works in .NET Framework 4.8.
In .NET 5, it works in x86 mode only.

Known Workarounds

No response

Configuration

• My app is currently on Framework 4.7.2; but code still works on 4.8; trying to run the code on .NET 7.
• Windows 10 Pro x64 22H2

Other information

No response

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[MichalStrehovsky]

GetSysColor is marked SuppressGCTransition:

runtime/src/libraries/Common/src/Interop/Windows/User32/Interop.GetSysColor.cs

Line 17 in c2ec86b

[SuppressGCTransition]

There are a lot of limitations on what such p/invoke can do. The hook is likely violating the contract.

[FenPhoenix]

Well that solves the mystery at least, thanks. Is there anything I can do? Should I take this up with one of the hooking libraries' authors?

[jkotas]

You can bring it up with them, but I do not think they will be able to do anything about it. It is not technically possible to safely hook OS APIs that are called by the runtime when the GC mode transition is suppressed.

[FenPhoenix]

Fair enough, thanks for the info.

[RussKie]

What was the reason for suppression? It feels like it regressed various customer scenarios.

[jkotas]

What was the reason for suppression?

See #37284

In general, we make no guarantees that it is possible to detour any OS APIs back into managed code. Many OS APIs are called in cooperative GC mode by the runtime and detouring them back into managed code is going result into a crash. The exact set is implementation detail and changing every release.

[AaronRobinsonMSFT]

Closing this as "by-design"