Multiple consumers with same group id get same message

[dfdx]

Kafka manual says that each message is delivered exactly to one consumer from a group (with a same group id). And this is what I see with Java high-level API and expected to see with Python's SimpleConsumer. However, when I run 2 consumers simultaneously (see code below) and send new message, both instances of consumer receive it.

---

# consumer.py
from kafka.client import KafkaClient
from kafka.consumer import SimpleConsumer

kafka = KafkaClient("broker", 9092)
consumer = SimpleConsumer(kafka, "my-group", "my-topic",
                          auto_commit=False)
while True:
    msg = consumer.get_message()
    if msg:
        print(msg)
    else:
        print('no new messages')
kafka.close()

---

# producer.py
from kafka.client import KafkaClient
from kafka.producer import SimpleProducer

kafka = KafkaClient("broker", 9092)

producer = SimpleProducer(kafka, "my-topic")
producer.send_messages("my message")

kafka.close()

---

Note, that auto_commit in consumer is set to False to overcome problem with Kafka 0.8.1 and current version of kafka-python, but changing it to True only adds exceptions and doesn't fix consumption problem.

[mumrah]

That contract of one message per consumer group only works for the coordinated consumers which are implemented for the JVM only (i.e., Scala and Java clients).

Kafka will eventually have better support for non-JVM clients to handle coordinated consumption at which point we'll implement this functionality.

HTH

[chitrakojha]

Hi Mumrah - we're using this with storm and want to use storm's parallelization (all storm spouts will be in the same consumer group for a topic). Do you know if this is possible to do through this ?

[wizzat]

Can you configure them to pull from a single partition? If so, yes. If not, no. There is a pull request which looks like it will provide that functionality once Kafka server 0.8.2 goes live.

On May 14, 2014, at 20:00, chitrakojha [email protected] wrote:

Hi Mumrah - we're using this with storm and want to use storm's parallelization (all storm spouts will be in the same consumer group for a topic). Do you know if this is possible to do through this ?

—
Reply to this email directly or view it on GitHub.

[gregorg]

Link to this PR : #164

[coffenbacher]

Is #164 still the answer for this one? Looks like it is still open / possibly abandoned. @wizzat is it expected to work out of the box on a single partition? I'll test it out, just wondering what the expected behavior is.

This is pretty important for my team, so if there's no working solution yet I may take a shot at adding this

[magcius]

It seems #164 was closed with "it was fixed with other commits" -- is that the case? We're still not seeing consumer balancing with kafka-python.

[magcius]

Oh, it seems this will be implemented in #38 -- this is a dupe of that, basically.

[dpkp]

yes, #38 is the tracking issue. consumer rebalancing will be supported when using kafka brokers >= v0.9.0.0 . Development is on the 0.9 git branch.

[sadaf2605]

pip3 freeze | grep kafka
kafka==1.3.4.1

All my consumer with same group are getting all the same messages from all the producers.