Etcd Client Sends Invalid Authority Header

[jeff-hegerhorst-deepsee]

Overview
Operator Image Version: milvusdb/milvus-operator:v0.7.13

The etcd client being used by the operator is sending an invalid authority header when trying to connect to etcd. This causes issues when etcd is being hosted behind a proxy like Istio because the sidecar rejects the request. This results in the health check of the etcd cluster to fail with the following logs:

{"level":"warn","ts":"2023-06-27T14:48:14.903Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invok
er failed","target":"etcd-endpoints://0xc000890540/#initially=[ds-milvus-etcd.default:2379]","attempt":0,"error":"rpc error: code = Unavailable de
sc = error reading from server: EOF"}
{"level":"warn","ts":"2023-06-27T14:48:15.933Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invok
er failed","target":"etcd-endpoints://0xc000890540/#initially=[ds-milvus-etcd.default:2379]","attempt":1,"error":"rpc error: code = Unavailable de
sc = error reading from server: EOF"}
{"level":"warn","ts":"2023-06-27T14:48:16.963Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invok
er failed","target":"etcd-endpoints://0xc000890540/#initially=[ds-milvus-etcd.default:2379]","attempt":2,"error":"rpc error: code = Unavailable de
sc = error reading from server: EOF"}
{"level":"warn","ts":"2023-06-27T14:48:17.993Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invok
er failed","target":"etcd-endpoints://0xc000890540/#initially=[ds-milvus-etcd.default:2379]","attempt":3,"error":"rpc error: code = Unavailable de
sc = error reading from server: EOF"}
{"level":"warn","ts":"2023-06-27T14:48:18.899Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invok
er failed","target":"etcd-endpoints://0xc000890540/#initially=[ds-milvus-etcd.default:2379]","attempt":4,"error":"rpc error: code = DeadlineExceed
ed desc = context deadline exceeded"}
2023/06/27 14:48:18 condition not ready EtcdReady False EtcdNotReady All etcd endpoints are unhealthy:[ds-milvus-etcd.default:2379:context deadlin
e exceeded]

Solution
The etcd client for the operator needs to be updated to a version greater than 3.5.0 see etcd-io/etcd#13192

[haorenfsa]

We changed default etcd version to 3.5.5