-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support and provide producer message timestamps #678
Conversation
- Support v0/v1/v2 of Produce requests in the protocol layer. - Enforce the correct message version for Kafka version at the Broker. - Use v2 messages in the producer if the Kafka version supports it, and pass the resulting timestamp back to the user in the ProducerMessage.
Hello @eapache et al, I have pulled this down and tried to use it, I assume it's enough to set the new version onto my config like so
However attempts to send a message fail with panic: runtime error: invalid memory address or nil pointer dereference goroutine 771 [running]: Changing the version to previous values works but the timestamp is not set |
Some more details I collected, This could potentially be a race between the goroutines started at broker.go:64 which sets the conf on the broker instance and async_producer.go:539 which calls broker.Produce as adding some printfs tells me b.conf is only nil for the first of many messages I am trying to send. Note the code in the stack trace above is not run for any other version. Or it could be that I need to initialize things some other way before I start producing. |
Yes, I believe it is a race, I have a fix in the works. |
The `Timestamp` field was introduced in IBM#678 and at this point was only set by the broker. Subsequent to this, the changes described in [KIP-32] were implemented, allowing the `Timestamp` of the message to be set by a producer when the broker was configured to use `CreateTime`. This commit updates the doc on the ProducerMessage struct to reflect that depending on which of the `CreateTime` and `LogAppendTime` modes are active, the `Timestamp` can be meaningfully set by either the request or the response. [KIP-32]: https://cwiki.apache.org/confluence/display/KAFKA/KIP-32+-+Add+timestamps+to+Kafka+message
The `Timestamp` field was introduced in IBM#678 and at this point was only set by the broker. Subsequent to this, the changes described in [KIP-32] were implemented, allowing the `Timestamp` of the message to be set by a producer when the broker was configured to use `CreateTime`. This commit updates the doc on the ProducerMessage struct to reflect that depending on which of the `CreateTime` and `LogAppendTime` modes are active, the `Timestamp` can be meaningfully set by either the request or the response. [KIP-32]: https://cwiki.apache.org/confluence/display/KAFKA/KIP-32+-+Add+timestamps+to+Kafka+message
The `Timestamp` field was introduced in IBM#678 and at this point was only set by the broker. Subsequent to this, the changes described in [KIP-32] were implemented, allowing the `Timestamp` of the message to be set by a producer when the broker was configured to use `CreateTime`. This commit updates the doc on the ProducerMessage struct to reflect that depending on which of the `CreateTime` and `LogAppendTime` modes are active, the `Timestamp` can be meaningfully set by either the request or the response. [KIP-32]: https://cwiki.apache.org/confluence/display/KAFKA/KIP-32+-+Add+timestamps+to+Kafka+message
resulting timestamp back to the user in the ProducerMessage.
Addresses #642
@makkes @wvanbergen