Bug Report: throttler: gRPC logs indicate attempts to contact a decomissioned IP address

[shlomi-noach]

Overview of the Issue

With the introduction of gRPC based throttler checks, and on a k8s environment, and given a specific rollout flow, we've seen errors (warnings) coming from the grpc-go package, like so:

component.go:41] [core] [Channel #652 SubChannel #653] grpc: addrConn.createTransport failed to connect to {Addr: "10.10.10.20:15999", ServerName: "10.10.10.20:15999", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.10.10.20:15999: i/o timeout"  

The errors are coming from the primary tablet. The IP address is that of a decomissioned replica. grpc-go seems to still "remember" that IP and routinely keeps polling it.

Observations:

• Disabling & enabling the throttler resolves the issue.
• If the throttler starts disabled, then enabled after the old tablet was decomissioned, the errors do not show.
• The error trace does not originate directly by throttler code, but rather from grpc-go routine keepalive or cleanup code.

The issue is already identified, PR to follow.

cc @vmg @deepthi

Reproduction Steps

Binary Version

v18

Operating System and Environment details

-

Log Fragments

No response