Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[QUESTION] idempotent producer and OutOfOrderSequenceNumber problem #1017

Open
funkindy opened this issue Jun 25, 2024 · 1 comment
Open
Labels

Comments

@funkindy
Copy link

I have two instances of aiokafka producers with enable_idempotence=True. Each instance is producing messages using only send_and_wait method. Once in a while they start to throw OutOfOrderSequenceNumber and producing messages stops until we restart pods.

Before this i see server logs, looks like cluster become unavailable on a short period of time:

Controller 1's connection to broker was unsuccessful (kafka.controller.RequestSendThread) java.io.IOException: Connection to host failed. at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:70) at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:298) at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:251) at org.apache.kafka.server.util.ShutdownableThread.run(ShutdownableThread.java:127)"

Connection to node could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)",

After that something reconnects and this error appears:

[ReplicaManager broker=2] Error processing append operation on partition merchants-0 (kafka.server.ReplicaManager) org.apache.kafka.common.errors.OutOfOrderSequenceException: Out of order sequence number for producer 29003 at offset 1641969 in partition merchants-0: 158 (incoming seq. number), 155 (current end sequence number)",

How is it possible that on the client side there is 158 sequence number, which is greater than the state on the broker (155). No messages were delivered.

Maybe someone can explain in details how idempotent producers work in aiokafka and how do they sync their states?

Thank you.

@LuWao
Copy link

LuWao commented Sep 3, 2024

+1, observe same behavior on configuration:

  • kafka: version: "3.1.2"
  • zookeeper: "3.7.1"
  • aiokafka: "0.8.1"

But in my case 2 of 3 zookeepers was dead, brokers continued to function.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants