Possible tail-latency regression from v2.0.0-beta5 -> v2.0.1

[ja30278]

🐛 Bug Report

Upgrading from v2.0.0-beta5 to v2.0.1, I noticed quite a large regression in tail latency performance.

I'm using grpc-gateway to front a localhost GRPC server in the same process, and p99 latency for my test workload increased from 85ms to 300ms. Rolling back grpc-gateway to v2.0.0-beta5 resolved the issue.

I profiled the server using pprof, but did not see obvious changes in the profile that would explain the increase.

To Reproduce

Build with v2.0.0-beta 5, send requests using ab (or other load generation tool), update grpc-gateway to v2.0.1, repeat

Expected behavior

Tail latency unchanged

Actual Behavior

Increase in p99 latency that was resolved by rolling back grpc-gateway to 2.0.0-beta5

Your Environment

Linux version 5.4.0-1031-azure (buildd@lcy01-amd64-021) (gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu118.04)) #3218.04.1-Ubuntu SMP Tue Oct 6 10:03:22 UTC 2020

[johanbrandhorst]

Thanks for the bug report! This is the first time I've ever seen a performance bug report, cool! Here are the changes between v2.0.0-beta.5 and v2.0.1: 462ce8c...5c1639c.

I'll have a look tomorrow to see if I can see anything obvious.

[ja30278]

@johanbrandhorst sorry for the sparse details and lack of a repro test case. I was trying to at least file the issue so that I can reference it in a PR to pin our version back to 2.0.0-beta.5. I'll see if I can narrow it down, I did have a look at the set of changes between the versions, and nothing obvious jumped out.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.