Starting a standalone consumer

[arjun289]

Hi,

I have started using kafka_ex for one of my IoT projects.
I have a requirement for standalone consumer which is not placed inside any consumer group. There is support in Kafka for the same, however in the codebase or docs I did not come across any such implementation.
The GenConsumer requires a consumer_group to start the process.

Am I missing anything?

[dantswain]

Hi @arjun289, you should be able to just call fetch in a loop for each partition.

[arjun289]

Thanks for your suggestion @dantswain.

The real issue here is that there are stream processing chains implemented with flow-based programming. All these chains belong to a specific project which group these chains.
Each time the user creates a chain it subscribes to a specific topic in Kafka depending on the project. The topic is basically telemetry which is created per project.
All the chains in the project subscribe to this topic, hence they should be individual consumers without any group as every chain should receive every event coming to the topic and will process it in its own way.

As per the above suggestion, I will be creating a worker per chain which would call the fetch. However, will it take care of semantics that is supported for GenConsumer such as offset management?

[dantswain]

Ah. Maybe I didn’t understand your requirement. You can manually commit the offsets. However, I think you may be able to just use consumer groups with a single member. The cluster should just assign all partitions to that consumer, and then offset management should work as expected.

[arjun289]

Thanks, @dantswain I think I will go ahead with the consumer group suggestion as it reduces efforts on my side.

[arjun289]

Will it be helpful if I raise a PR with implementation to handle standalone consumer?

[dantswain]

Thanks for the offer @arjun289 I think your use case is not really outside the intended use case for consumer groups, so I’m not sure what value there would be in having another implementation. On the other hand, there are a few benefits to using the consumer group mechanism here. For example, if you accidentally start a duplicate consumer there will be no double-consuming of messages.

[cdesch]

@arjun289 the consumer_group in the config can be set to :no_consumer_group

Starting a individual consumer can be done like this:

KafkaEx.GenConsumer.start_link(
  Cx.ExampleGenConsumer,
  "foo", 
  "foo",
  0,
  commit_interval: 5000,
  commit_threshold: 100
)

Not sure about the supervisor yet.