cdc: test Kafka network connectivity from nodes

[shermanCRL]

Is your feature request related to a problem? Please describe.

Changefeed users may have network connectivity problems from CRDB to Kafka. This is difficult to troubleshoot. The error returned is often kafka: client has run out of available brokers to talk to (Is your cluster reachable?), which I understand to result from any network problem.

Further, Cloud users do not have network troubleshooting tools (ping, nslookup, traceroute) because they can’t ssh into Cloud nodes. And, they don’t have access to logs, where they might cat *.log | grep sarama.

Problems might also include certs and auth.

Describe the solution you'd like

The ability to test Kafka connectivity from the CRDB process, via the SQL client. The Disaster Recovery team has implemented similar for cloud storage: #97517

The solution may be an extension of the above to accept kafka:// URLs. We’d need some way to generate Kafka-specific workload.

See also kcat for inspiration: https://docs.confluent.io/platform/current/clients/kafkacat-usage.html

We might go further and add things like WITH dns, traceroute, ping etc, which would not be Kafka-specific. And then iterate on other sinks such as webhook and pubsub.

---

Possbile duplicate of #60178

Jira issue: CRDB-25245
Epic CRDB-6667

[blathers-crl]

cc @cockroachdb/cdc