KafkaConsumer always passes message key to deserializer, even if key is None

[jeffwidman]

I specified json.loads as the KafkaConsumer's key_deserializer callable, then produced a message without a key.

json.loads throws an exception about expecting a string. Looks like root cause is here: https://github.com/dpkp/kafka-python/blob/master/kafka/consumer/fetcher.py#L546

Fetcher is always using passing the key to the deserializer, even if the key is None.

Will submit a PR shortly.

[dpkp]

We generally wrap json.loads in a lambda to check for None. But I don't think we can require that all deserializers map None -> None, however, so we do not attempt to short circuit or special case internally.

…

On Dec 18, 2016 3:50 PM, "Jeff Widman" ***@***.***> wrote: I specified json.loads as the KafkaConsumer's key_deserializer callable, then produced a message without a key. json.loads throws an exception about expecting a string. Looks like root cause is here: https://github.com/dpkp/kafka-python/blob/master/kafka/ consumer/fetcher.py#L546 Fetcher is always using passing the key to the deserializer, even if the key is None. Will submit a PR shortly. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#913>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAzetL_mOjJXDW2OLSh2JfRVdMteWxfZks5rJcangaJpZM4LQRat> .

[jeffwidman]

Thanks for the fast reply.

I don't think we can require that all deserializers map None -> None

That makes sense.

We generally wrap json.loads in a lambda to check for None.

Good suggestion, hadn't thought of that. I'm guessing you mean lambda k: json.loads(k) if k is not None else k unless there's something special that I'm unaware of regarding how python's lambdas deal with None?

[jeffwidman]

Are you open to adding a note about this lambda tip to the tutorial doc, so that others don't have to debug it?

There's an example showing how to use the json.loads already, so just add another param on this line showing how to do this for keys with a comment like

 # using a lambda because null keys are expected, but json.loads throws exception if key=None

[dpkp]

Yes -- exactly. Although I agree that it is much more common to want None -> None, and much less common to want something else, I dont think we should make it impossible to do the latter. Options I think are ok: improve docs, provide a "stock" json deserializer, add a configuration to disable None skipping.

Also you might want to take a quick look at #912 where I'm changing the internals of serializer / deserializer handling slightly.