Wrong exception for large messages.

[sudev]

Hello all,

With a balanced/simple consumer if you are trying to fetch messages larger than default fetch_message_max_bytes value, I'm getting a exception NoMessagesConsumedError.

2016-03-11 14:45:12,291.291.701078415:pykafka.simpleconsumer:139956779079424:ERROR:11706:Exception encountered in worker thread:
  File "/usr/local/python/lib/python2.7/site-packages/pykafka/simpleconsumer.py", line 346, in fetcher
    self.fetch()
  File "/usr/local/python/lib/python2.7/site-packages/pykafka/simpleconsumer.py", line 696, in fetch
    min_bytes=self._fetch_min_bytes
  File "/usr/local/python/lib/python2.7/site-packages/pykafka/broker.py", line 244, in fetch_messages
    return future.get(FetchResponse)
  File "/usr/local/python/lib/python2.7/site-packages/pykafka/handlers.py", line 66, in get
    return response_cls(self.response)
  File "/usr/local/python/lib/python2.7/site-packages/pykafka/protocol.py", line 731, in __init__
    partition_id=partition[0]),
  File "/usr/local/python/lib/python2.7/site-packages/pykafka/protocol.py", line 738, in _unpack_message_set
    message_set = MessageSet.decode(buff, partition_id=partition_id)
  File "/usr/local/python/lib/python2.7/site-packages/pykafka/protocol.py", line 326, in decode
    raise NoMessagesConsumedError()

Once I increased the fetch_message_max_bytes value consumer was able to consume messages properly, so the exception should have been ideally MessageSizeTooLarge and not NoMessageConsumed.

PyKafka version: latest
Kafka version: 0.8.2

Thanks for making Pykafka!

[emmettbutler]

Thanks @sudev - I'm glad you're getting value out of pykafka. I agree that the exception here should be MessageSizeTooLarge. Making that change isn't actually trivial (and perhaps impossible, but I'm not certain), so I'll have to dig in a bit. I think it's worthwhile, though.

My concern is that there may be other cases where this condition is met aside from fetching messages that are too large. I'll have to pick through the code to determine whether or not that's the case.