Upstream gRPC connection reset when missing TE header value

[hq6]

Title: Upstream gRPC connection reset when missing TE header value

Description:

Observed behavior: When running grpcurl using Envoy as a proxy to a Python gRPC client, the upstream server resets the connection.

ERROR:
  Code: Unavailable
  Message: upstream connect error or disconnect/reset before headers. reset reason: remote reset

When examining the network traffic using Wireshark, I observed that Envoy has stripped out the TE: trailers header.

When examining the verbose output of the Python gRPC server, I observe the following logs complaining about the missing header.

I0603 03:20:17.375027272    3057 chttp2_transport.cc:1646]   perform_stream_op[s=0x7f358c008160]:  CANCEL:{"created":"@1717384817.375006652","description":"Missing :authority or :path","file":"src/core/lib/surface/server.cc","file_line":1420,"referenced_errors":[{"created":"@1717384817.375004921","description":"Failed processing incoming headers","file":"src/core/ext/filters/http/server/http_server_filter.cc","file_line":123,"referenced_errors":[{"created":"@1717384817.375003392","description":"Missing header","file":"src/core/ext/filters/http/server/http_server_filter.cc","file_line":204,"key":"te"}]}]}

Expected Behavior:

• We should observe a response from the application gRPC handler.
• For gRPC, we should observe that the TE header is preserved, as required by the gRPC protocol.

Notes:

• I see that d4ed8d4 by @alyssawilk was intended to fix Upstream H2 connection reset when sending invalid TE header value #30362
• If it is possible to special case that commit to preserve the TE header or ensure that the TE header is present when the upstream is a gRPC server, I think that would fix this issue.
• The te header requirement appears to be enforced by at least the C-based gRPC implementations, so I suspect that others will run into this issue independently if the default behavior is not changed.
• We can workaround this for now by disabling envoy.reloadable_features.sanitize_te, but that means the flag has to stick around until this issue is resolved.

[alyssawilk]

#32255 should have addressed this - please check it out!

[hq6]

#32255 should have addressed this - please check it out!

Thanks!
Looking at the code, it looks like it should address this issue, so we can close this issue for now.
I can test it after Block picks up this change in our internal Envoy deployment.