ConsumerGroup.Manager does not proxy along worker options

[zapient]

We are trying to configure our producers and consumer groups with individual cluster configurations but we noticed that in our review environments that use SSL we were unable to connect.

Supervisor.Spec.supervisor(
  KafkaEx.ConsumerGroup,
  [
    OurSweetConsumer,
    "my-topic",
    ["my-consumer-group-name"],
    [uris: [{"ec2-xxx.compute-1.amazonaws.com", 9096}], use_ssl: true, ssl_options: [cert: <<...>>, key: { :RSAPrivateKey, <<..>>}]]
  ]
)

This is because ConsumerGroup.Manager is dropping all but the uris option.
I'm curious if this was by design and that I am totally missing the reasoning behind it.

Note the same issue is on GenConsumer

[joshuawscott]

I don't know if this was an intentional decision, but you can handle this by creating your own worker(s) and passing it into the fetch_options: of a consumer group.
So,

{:ok, _} = KafkaEx.create_worker(MyWorker, uris: [{"ec2-xxx.compute-1.amazonaws.com", 9096}], use_ssl: true, ssl_options: [cert: <<...>>, key: { :RSAPrivateKey, <<..>>}])
Supervisor.Spec.supervisor(
  KafkaEx.ConsumerGroup,
  [
    OurSweetConsumer,
    "my-topic",
    ["my-consumer-group-name"],
    [fetch_options: [worker_name: MyWorker]]
  ]
)

Edit:
I think we'd be open to something to make this easier, although I would suggest the easiest way to handle this might be to allow passing in a worker and ensuring it uses that worker for all communication.

[zapient]

Any objections to a simple addition to include ssl and ssl_options to the proxied worker_opts?

[markusfeyh]

To close the loop here, added a PR to add this here: #413

It ends up requiring less verbosity and allows for passing through options to the worker @joshuawscott

[markusfeyh]

This can be closed now that the PR has been merged in

[joshuawscott]

Thanks!