suboptimal load balancing of veneur-proxy in kubernetes

[singron]

If you set up veneur-proxy as a Service in kubernetes, then kubernetes will expose that as a single IP address that will transparently assign connections to the actual hosts. E.g. see the part about virtual IPs in this article.

Unfortunately, the grpc client libraries are completely unaware of this and will send all requests over 1 TCP connection. If any of the hosts go down, these singleton TCP connections will be assigned to one of the remaining available hosts. Eventually this can lead to situations where all clients are connecting to the same veneur-proxy instance despite multiple being available.

In our fork, I integrated with https://github.com/sercand/kuberesolver so that grpc would round robin among the available veneur-proxy instances. I guess alternatively you could use some type of proxy sidecar like linkerd, but that isn't part of our stack. I'm curious if others are running into this problem and what types of solutions you would accept upstream.