Release 0.11.0

[joshuawscott]

I cleaned up a couple of things in the README that were out of date.

Proposed release notes follow.

KafkaEx 0.11.0 is a large improvement to KafkaEx that sees the introduction of the Kayrock client, numerous stability fixes, and a critical fix that eliminates double-consuming messages when experiencing network failures under load.

Breaking changes

• Drop support for Elixir < 1.5
• Partitioner has been fixed to match the behavior of the Java client. This will cause key assignment to differ. To keep the old behavior, use the KafkaEx.LegacyPartitioner module. (Change default KafkaEx partitioner to correctly match the Java client #399)

Fixes

• Numerous fixes to error handling especially for networking connections (Attempt of better connection error handling / reconnect #347, Close socket on timeout/error #351 )
• Metadata is refreshed after topic create/delete(Refresh metadata after creating or deleting a topic #349)
• Compression actually works for producing now 😬 (Honor compression option on produce #362)
• Don't crash when throttled by quotas (Fix quota crash #402)
• Default partitioner works the same way as the Java client now (Change default KafkaEx partitioner to correctly match the Java client #399)
• When fetching metadata for a subset of topics, don't remove the metadata for the unfetched topics. (Do not update metadata cache on metadata call #409)

Improvements

• Any GenServer can be used as a KafkaEx.GenConsumer (Allow any GenServer to act as a consumer managed by ConsumerGroup. #339)
• Can specify wait time before attempting reconnect (Attempt of better connection error handling / reconnect #347)
• KafkaEx.stream/3 now uses the KafkaEx.Server interface, which inherits the sync_timeout setting. This avoids timeouts when sync_timeout needs to be greater than default. (Make KafkaEx.stream/3 use KafkaEx.Server #354)
• KafkaEx can now use Kayrock as the client implementation. This is a large change, and is pushing toward the improvements we want in 1.0 to allow the library to easily support new versions of the Kafka protocol. (Kayrock integration - Compatibility with existing API up to Consumer Group usage #356, Handle consumer groups with new client #359, Allow specifying API version for produce/fetch with new client, support timestamps #364, Allow specifying versions for offset commit and fetch #366, Support setting api versions for consumer groups #367, Allow specifying kafka API versions for streaming #369, Allow setting the offset commit api version on fetch requests #370, Respect max kayrock version when determining max usable api version #374, Allow consumer group to retry join #375, Version bump Kayrock #377, Fix default partitioner for kayrock #379, Fix kayrock auto commit behavior on empty results #387, Handle closed sockets in new client #406, Fix socket close error #408, )
• KafkaEx.Protocol.Fetch.Message includes the topic and partition, allowing consumers to know which topic and partition they consumed from.
• Add KafkaEx.start_link_worker/1-2 to start a working and link it to the current process.
• Allow setting the client_id for the application - supports better monitoring, debugging, and quotas. (Configurable client id #388)
• Send metadata requests to a random broker rather than the same one each time (Don't always send update metadata requests to the same broker #395)
• Retry joining a consumer group 6 times rather than failing (Allow consumer group to retry join #375, Increment attempt number instead of decrement when joining a consumer group #403)

Kayrock client

See Kayrock and the Future of KafkaEx

Note that the Kayrock implementation doesn't support Kafka < 0.11

Improvements over default client:

• Can specify message API versions for the KafkaEx.stream/3 API
• Can specify message API versions for the consumer group API - NOTE that if you specify OffsetCommit API version >= 2, it will attempt to store the offsets in kafka, which will have no data about the topic unless you migrate the data. This could result in losing or reprocessing data. Migration can be done out of band, or can be done via the appropriate API calls within KafkaEx.
• Allow specifying OffsetCommit API version in KafkaEx.fetch

Misc

• Test suite uses KafkaEx 0.11 now in preparation for fully supporting Kafka API versions.
• KafkaEx.Protocol.Produce cleaned up (Translate int size values into module attributes #380)
• Documentation improvements (correct minor typo #383, Fix broken README links to Kayrock and new API #384)
• Test against Elixir 1.9 (test other branch #394)
• Use OTP 22.3.3+ for OTP 22.2 testing to avoid SSL bug. (Timeout errors: ensure version of OTP in CI + document in README #405)

Thanks

Thanks to @gerbal, @ukrbublik, @jbruggem, @thulio, @dantswain, @thiamsantos, @kennethito, @habutre, @davidrusu, @getong, @shamilish, @adrienmo, @mjparrott, @richardlarocque for their contributions!

[bjhaid]

Great job! thanks for the well written release note.