question about consumer group.

[xiaojun789]

in the kakfa document said, only one conusmer can consume one partition in the same consumer group. but I test kafka-node, it seems allow doing this.

doese the kafka-node support this feature? or I have some mistake? Thanks~!

[haio]

I think multiple consumers of the same consumer group can consume the same one partition, as is kafka-node.There is no way to prevent user from creating many consumer with the same group, it's all up to you. The default consumer group of consumer in kafka-node is kafka-node-group.

[geoffrey-young]

from

from https://kafka.apache.org/documentation.html#introduction"

"Consumers label themselves with a consumer group name, and each message published to a topic is delivered to one consumer instance within each subscribing consumer group. Consumer instances can be in separate processes or on separate machines. If all the consumer instances have the same consumer group, then this works just like a traditional queue balancing load over the consumers."

kafka-node doesn't currently support this, as far as I can tell - concurrent kafka-node scripts using the same groupId will all receive the same message, and therefore each message is processed multiple times.

this comment:

dpkp/kafka-python#112

makes me think it's not possible, but I haven't been able to figure out why some languages are capable of supporting coordinated consumers and others are not...

[haio]

This problem is interesting, I don't know why this feature not support non-JVM clients, maybe kafka guys can explain it ^_^.

[karthic891]

Hi haio,

As @geoffrey-young pointed out, in Kafka, message distribution is load balanced across all the consumers of the same consumer group and hence should not receive the same message.

Currently with kafka-node, all the consumers under a consumer group, listening to the same topic, receive the same set of messages. Would there be a fix for this in kafka-node?

I'm building a Node.js + Socket.io cluster for server-side push using Kafka as the event-driven messaging system. The kafka-node consumers in all the nodes in my cluster is listening to the same topic and gets the same message, and hence I end up pushing the same message to the browser multiple times(the same number as the number of Kafka consumers).

Thanks & Regards,
Karthic

[dmtrs]

+1

[dmtrs]

@karthic891 Did you actually resolve this issue?

[karthic891]

@dmtrs No, the issue is still open as I couldn't find any alternative way(hack) to achieve the desired behaviour. CC'ing @haio

[dmtrs]

@haio Can we reopen this issue please. It seems that they have result issue in python client

[rajiff]

+1

Let's try to see how this can be fixed, if possible, I am willing to contribute with some help

[hyperlink]

Are you using the ConsumerGroup? This is not still an issue afaik.