Evaluate Confluent Kafka client

[matzew]

Currently both, channel and source are using the Sarama library for producer/consumer parts.

Ideally we use a more improved, high-level, API like the kafka lib from Confluent (Apache2 licensed). but the CGo dependency might be an issue...

[matzew]

With the regard of DQL support, this lib is also "interesting":

https://github.com/uber-go/kafka-client

but I have no idea on the actual PERF

[lionelvillard]

The DQL support seems compatible with our design, one DLQ per topic and allowing DLQs to be on different clusters.

This API is alpha though....

[matzew]

@lionelvillard right, that's exactly why I put the DQL comment in here :-)

[matzew]

related: #596

[slinkydeveloper]

IMHO I think sarama works well as a library. I think we should however improve our usage of it. On example could be: move message dispatching outside the goroutines that handles the partition processing

[matzew]

+1 on that!

On Tue 17. Sep 2019 at 10:06, Francesco Guardiani ***@***.***> wrote: IMHO I think sarama works well as a library. I think we should however improve our usage of it. On example could be: move message dispatching outside the goroutines that handles the partition processing — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#588?email_source=notifications&email_token=AABGPTR5ZUDS73DCHPDE5WDQKCFZPA5CNFSM4IWNZJY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD63V22A#issuecomment-532110696>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABGPTRQSMGEO5WP5C62WKLQKCFZPANCNFSM4IWNZJYQ> .

-- Sent from Gmail Mobile

[haneefkassam]

The other problem with the Confluent library is the external dependency on librdkafka, the installation of which requires some dependencies. Sarama being pure go alleviates some of those concerns.

[matzew]

yep, thats part of the problem

On Wed 25. Sep 2019 at 16:15, haneefkassam ***@***.***> wrote: The other problem with the Confluent library is the external dependency on librdkafka, the installation of which requires some dependencies. Sarama being pure go alleviates some of those concerns. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#588?email_source=notifications&email_token=AABGPTWPX54ICE5ZMKXNEV3QLNW7PA5CNFSM4IWNZJY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7SCAKQ#issuecomment-535044138>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABGPTU7PVIVDDHHFG3JJITQLNW7PANCNFSM4IWNZJYQ> .

-- Sent from Gmail Mobile

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[pierDipi]

We evaluated and decided to use sarama.
/close

[knative-prow-robot]

@pierDipi: Closing this issue.

In response to this:

We evaluated and decided to use sarama.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.