Kafka Configuration Improvements

[jeastham1993]

Currently, a Kafka broker endpoint is configured in the configuration file, but if credentials are required then they need to be passed in as part of the subscription. Update this to allow credentials to be specified at the top level, keeping the subscription calls simple.

Additional Context

I'd want to configure everything related to my Kafka connection at a higher level, in some kind of configuration. And then my subscriptions can stay exactly the same regardless of which server I'm connecting to
In the current model, if it works as you suggest, when I'm developing locally Vs when I'm deploying to a 'real' Kafka cluster I'd need to change all my subscriptions...
Whereas really I want to be able to switch my connection around (local, dev, prod etc) any the subscriptions 'just work'

Proposed Implementation

Update the ingress.kafka-clusters.brokers config option to take a struct instead of an array of strings. Allow the sasl.username and sasl.password to be specified as part of this config object

[slinkydeveloper]

Hi, If I understood correctly you just want a different set of cluster setups right?

You should be able to do it already as follows:

[[ingress.kafka-clusters]]
name = "my-cluster-1"
brokers = ["PLAINTEXT://localhost:9092"]
sasl.username = "bla"
sasl.password = "very-secret"

[[ingress.kafka-clusters]]
name = "my-cluster-2"
brokers = ["PLAINTEXT://localhost:9093"]
sasl.username = "another-bla"
sasl.password = "very-secret-2"

And then when creating subscriptions, simply refer to the cluster configuration using name:

curl localhost:9070/subscriptions -H 'content-type: application/json' \
    -d '{
        "source": "kafka://my-cluster-1/my-topic",
        "sink": "service://MyService/handleCluster1",
        "options": {"auto.offset.reset": "earliest"}
    }'

curl localhost:9070/subscriptions -H 'content-type: application/json' \
    -d '{
        "source": "kafka://my-cluster-2/my-topic",
        "sink": "service://MyService/handleCluster2",
        "options": {"auto.offset.reset": "earliest"}
    }'

brokers may be a misleading name, that's what kafka usually calls bootstrap.servers in the consumer config.

[jeastham1993]

@slinkydeveloper ah, so you can do it in the configuration file? It's un-documented, and I asked in the Discord and the response was that it had to be done in each individual subscription. Maybe it's just the documentation that needs updating then?

[slinkydeveloper]

ah, so you can do it in the configuration file?

Yep the ingress.kafka-clusters config entry is a list of objects, so it works fine with any number of kafka cluster configurations you want https://docs.restate.dev/operate/configuration/server#configuration-reference

Maybe it's just the documentation that needs updating then?

Yep you're right, looks like an area of our documentation that we need to improve, I'll sort this out :)