You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
With the introduction of gRPC based throttler checks, and on a k8s environment, and given a specific rollout flow, we've seen errors (warnings) coming from the grpc-go package, like so:
The errors are coming from the primary tablet. The IP address is that of a decomissioned replica. grpc-go seems to still "remember" that IP and routinely keeps polling it.
Observations:
Disabling & enabling the throttler resolves the issue.
If the throttler starts disabled, then enabled after the old tablet was decomissioned, the errors do not show.
The error trace does not originate directly by throttler code, but rather from grpc-go routine keepalive or cleanup code.
Overview of the Issue
With the introduction of gRPC based throttler checks, and on a k8s environment, and given a specific rollout flow, we've seen errors (warnings) coming from the
grpc-go
package, like so:The errors are coming from the primary tablet. The IP address is that of a decomissioned replica.
grpc-go
seems to still "remember" that IP and routinely keeps polling it.Observations:
grpc-go
routine keepalive or cleanup code.The issue is already identified, PR to follow.
cc @vmg @deepthi
Reproduction Steps
Binary Version
Operating System and Environment details
Log Fragments
No response
The text was updated successfully, but these errors were encountered: