Root cause of connection error not apparent from logs

[k-wall]

We were faced with an issue where the kafka had become configured with an invalid certificate. The canary was failing to connect, but the error message was vague. This meant that time to problem resolution was longer than necessary.

W1129 19:59:18.376296     117 main.go:120] Error creating new Sarama client, retrying in 5000 ms: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)
W1129 19:59:24.154411     117 main.go:120] Error creating new Sarama client, retrying in 10000 ms: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

Sarama logging needed to enabled in order for the root cause to become clear.

[Sarama] 2021/11/29 19:58:46 client/metadata fetching metadata for all topics from broker 172.30.189.70:9093
[Sarama] 2021/11/29 19:58:46 Failed to send SASL handshake 172.30.189.70:9093: x509: cannot validate certificate for 172.30.189.70 because it doesn't contain any IP SANs
[Sarama] 2021/11/29 19:58:46 Error while performing SASL handshake 172.30.189.70:9093
[Sarama] 2021/11/29 19:58:46 Closed connection to broker 172.30.189.70:9093
[Sarama] 2021/11/29 19:58:46 client/metadata got error from broker -1 while fetching metadata: x509: cannot validate certificate for 172.30.189.70 because it doesn't contain any IP SANs
[Sarama] 2021/11/29 19:58:46 client/metadata no available broker to send metadata request to
[Sarama] 2021/11/29 19:58:46 client/brokers resurrecting 1 dead seed brokers

It would be desirable if the strimzi-canary warning and error message could carry the root cause.

[k-wall]

I took a look at this. It looks impossible to do without changes to the sarama client to propagate the error from b.connErr in Broker.Open() back to the user of the sarama API. At the moment, it seems like the detail is lost.

Sarama uses sentinel error pattern, so unfortunately one cannot simply change Sarama to Go's 1.13 error wrapping without breaking user code that does something like if err == ErrOutOfBrokers (they would need to refactor to errors.Is)

Whilst this would still be a useful feature to have, #166 would be useful alternative.

[k-wall]

As the upgrade to the latest Sarama client includes IBM/sarama#2131, I believe this issue can be closed. @ppatierno