Subsequent Grpc calls with WinHttpHandler to an IIS hosted service fail

[angelyordanov]

Description

I'm trying to call a Grpc service hosted in IIS over http2 from NetFramework app. After reading the relevant docs(supported platforms) and issues (dotnet/aspnetcore#9020, dotnet/core#5713) I understand that this should be possible if I'm using Win11/WinServer2022 with the pre release version of the System.Net.Http.WinHttpHandler nuget package (in my case 6.0.0-rc.1.21451.13).

Unfortunately this seems not to be the case, as only the first call to the service succeeds and any subsequent call fails with Status(StatusCode="Cancelled", Detail="No grpc-status found on response. Using gRPC with WinHttp has Windows and package version requirements. See https://aka.ms/aspnet/grpc/netstandard for details.").

Things get even stranger when I tested combining calls made with the NetCore's SocketsHttpHandler. A call with the WinHttpHandler will fail no matter whether the first call to the service was made with WinHttp or Sockets, and it appears that the problem is somehow triggered by the state of the IIS application as restarting the host resets the issue(restarting the client has no effect). Furthermore even after the WinHttp call has failed one can successfully call the service with the Sockets handler from the same process.

Here is a table with the different setups

Client	Server	Result
WinHttp	IIS	❌ Fail
WinHttp	Kestrel	✔️ Succeeds
Sockets	IIS	✔️ Succeeds
Sockets	Kestrel	✔️ Succeeds

Reproduction Steps

I've created a minimal project that reproduces the problem - https://github.com/angelyordanov/grpc-iis-winhttp-sample

1. Start the GrpcService with the IISExpress profile
2. Start either one of the console apps

Expected behavior

There should be no exceptions and all grpc calls should succeed.

Actual behavior

The second call made with the WinHttpHandler always fails.

Regression?

No

Known Workarounds

No

Configuration

dotnet --info
.NET SDK (reflecting any global.json):
 Version:   5.0.401
 Commit:    4bef5f3dbf

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.22000
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\5.0.401\

Other information

No response

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

I'm trying to call a Grpc service hosted in IIS over http2 from NetFramework app. After reading the relevant docs(supported platforms) and issues (dotnet/aspnetcore#9020, dotnet/core#5713) I understand that this should be possible if I'm using Win11/WinServer2022 with the pre release version of the System.Net.Http.WinHttpHandler nuget package (in my case 6.0.0-rc.1.21451.13).

Unfortunately this seems not to be the case, as only the first call to the service succeeds and any subsequent call fails with Status(StatusCode="Cancelled", Detail="No grpc-status found on response. Using gRPC with WinHttp has Windows and package version requirements. See https://aka.ms/aspnet/grpc/netstandard for details.").

Things get even stranger when I tested combining calls made with the NetCore's SocketsHttpHandler. A call with the WinHttpHandler will fail no matter whether the first call to the service was made with WinHttp or Sockets, and it appears that the problem is somehow triggered by the state of the IIS application as restarting the host resets the issue(restarting the client has no effect).

Here is a table with the different setups

Client	Server	Result
WinHttp	IIS	❌ Fail
WinHttp	Kestrel	✔️ Succeeds
Sockets	IIS	✔️ Succeeds
Sockets	Kestrel	✔️ Succeeds

Reproduction Steps

I've created a minimal project that reproduces the problem - https://github.com/angelyordanov/grpc-iis-winhttp-sample

1. Start the GrpcService with the IISExpress profile
2. Start either one of the console apps

Expected behavior

There should be no exceptions and all grpc calls should succeed.

Actual behavior

The second call made with the WinHttpHandler always fails.

Regression?

No

Known Workarounds

No

Configuration

dotnet --info
.NET SDK (reflecting any global.json):
 Version:   5.0.401
 Commit:    4bef5f3dbf

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.22000
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\5.0.401\

Other information

No response

Author:	angelyordanov
Assignees:	-
Labels:	area-System.Net.Http, untriaged
Milestone:	-

[antonfirsov]

This exception message is coming from grpc-dotnet, I recommend to open an issue on that repo, unless @JamesNK can confirm that this is something we should address in WinHttpHandler.

[JamesNK]

The error is coming from gRPC, but the reason it is throwing that error is that it can't find trailers on the HTTP response from WinHttpHandler.

It's really weird that this only happens when WinHttp is combined with IIS.

[JamesNK]

Here is SocketsHttpHandler calling IIS:

Trailer grpc-status is correctly set.

Here is WinHttpHandler calling IIS:

Trailer grpc-status is not set.

For sanity, here is WinHttpHandler calling Kestrel:

Trailer grpc-status is correctly set.

So there is something about WinHttpHandler + IIS + trailing headers that doesn't work.

IIS requires TLS for HTTP/2 so I can't get a Wireshark trace but I'm pretty sure the server will always be outputting trailing headers so it is likely an issue in WinHttpHandler.

All tests on .NET 6.

.NET SDK (reflecting any global.json):
 Version:   6.0.100-rtm.21504.5
 Commit:    457785c87e

[JamesNK]

I think there is a race condition here.

WinHttpHandler + IIS occasionally correctly loads the trailers. 90% of the time it fails. 10% of the time it works.

My theory is there is something about the shape of HTTP/2 frames that IIS returns compared to Kestrel.

[Tratcher]

I wasn't able to repro this with IIS Express (VS 2022 RC) on Win11 22483.1011 & WinHttpHandler 6.0-rc2.

[antonfirsov]

I'm can reproduce this with Win 11 22000.318 + VS 2019.

In the second call to the service GetResponseHeaderCharBufferLength returns 0, because the underlying WinHttp call returns 0 in bufferLengthInBytes:

runtime/src/libraries/System.Net.Http.WinHttpHandler/src/System/Net/Http/WinHttpResponseParser.cs

Lines 272 to 278 in a0824ba

	bool result = Interop.WinHttp.WinHttpQueryHeaders(
	requestHandle,
	infoLevel,
	Interop.WinHttp.WINHTTP_HEADER_NAME_BY_INDEX,
	new IntPtr(buffer),
	ref bufferLengthInBytes,
	ref index);

This implies to me that WinHttp indeed thinks that there are no trailers in that response. Note that this happens comes consistently on the second call, I don't see how could there be a race condition.

Ideas:

• Check VS2022 IIS Express in the same environment.
• Update to a newer build of Win11 and/or ask the WinHttp team if they are aware of any recent WINHTTP_QUERY_FLAG_TRAILERS issue they fixed
• Packet capture + decrypt TLS (never done this before but should be possible with WireShark AFAIK ...)

[JamesNK]

Issue is in WinHttpHandler.

WINHTTP_OPTION_REQUIRE_STREAM_END needs to be specified to instruct WinHttp to read all the way to the end of the response (e.g. trailing headers frame). Without this option then WinHttp stops once it has read up to content-length.

Issue shows up when WinHttpHandler is used with IIS because IIS sometimes includes content-length header with response.

[antonfirsov]

Reopening to track 6.0.x backport. This is a bug blocking .NET Framework gRPC developer workflow in Visual Studio + IIS Express.

[antonfirsov]

Backport PR is merged, this should be fixed with the upcoming (6.0.2 ?) update of the WinHttpHandler package.

[karelz]

Fixed in 7.0 (main) in PR #62870 and in 6.0.2 in PR #63346 6.0.4 in PR #63346 and #65523 (which is WinHttpHandler NuGet version 6.0.1 to make versioning even more confusing)

[angelyordanov]

Now that .NET 6.0.2 is released shouldn't there be a 6.0.2 version for the System.Net.Http.WinHttpHandler nuget package with this fix included?

[JamesNK]

@karelz It doesn't appear that the NuGet - https://www.nuget.org/packages/System.Net.Http.WinHttpHandler/ - gets service releases. Is this expected? How should customers get this fix?

[karelz]

That is weird. @antonfirsov did we forget to update some package magic to get it released?

[antonfirsov]

Apologies, looks like we didn't follow the nuget servicing guideline so it missed the release. I'm sorry for this. Working on it, so we don't miss the train. Reopening to track that work.

[karelz]

Fixed in 7.0 (main) in PR #62870 and in 6.0.2 in PR #63346 6.0.4 in PR #63346 and #65523 (which is WinHttpHandler NuGet version 6.0.1 to make versioning even more confusing)