Docker image cannot connect to kafka(both in/out) using SASL/SCRAM #6072

phillui-37 · 2019-07-05T07:55:29Z

Relevant telegraf.conf:

# There are two configs which related to two telegraf instance having the same issue.
## kafka -> influx ##
[[outputs.influxdb]]
url = "http://influx_dest:8086"
database = "telegraf"

[[inputs.kafka_consumer]]
brokers = ["a1:9094","a2:9094","a3:9094"]
topics = ["test-topic"]
version = "0.11.0.0"
sasl_username = "test"
sasl_password = "wtever"
consumer_group = "test"
offset = "oldest"
max_message_len = 1000000
data_format = "influx"
insecure_skip_verify = true

## influx -> kafka ##
[[inputs.influxdb]]
urls = ["http://influx_src:8086/debug/vars"]
timeout = "5s"

[[outputs.kafka]]
brokers = ["a1:9094","a2:9094","a3:9094"]
version = "0.11.0.0"
topic = "test-topic"
sasl_username = "test"
sasl_password = "wtever"
data_format = "influx"
compression_codec = 3
insecure_skip_verify = true
required_acks = 1

System info:

System: OSX 10.14.5/Arch Linux 5.1.15
Docker: (OSX) 18.09.2, build 6247962/ (Arch Linux) 18.09.7-ce, build 2d0083d657
Telegraf: 1.11.1 (edge docker image)
librdkafka: 1.1.0

Steps to reproduce:

Start the two telegraf docker instances with the two configs.

Expected behavior:

Connection is build between influxdb and kafka. Data synchronization start automatically.

Actual behavior:

Error occur and instance crash. Instance cannot connect to kafka. Telnet direct connect test ok.

Additional info:

influx -> kafka

2019-07-05T07:38:39Z I! Starting Telegraf 1.11.1
2019-07-05T07:38:39Z I! Using config file: /etc/telegraf/telegraf.conf
2019-07-05T07:38:39Z I! Loaded inputs: influxdb
2019-07-05T07:38:39Z I! Loaded aggregators: 
2019-07-05T07:38:39Z I! Loaded processors: 
2019-07-05T07:38:39Z I! Loaded outputs: kafka
2019-07-05T07:38:39Z I! Tags enabled: host=612ddcc85b3e
2019-07-05T07:38:39Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"612ddcc85b3e", Flush Interval:10s
2019-07-05T07:38:53Z E! [agent] Failed to connect to output kafka, retrying in 15s, error was 'kafka: client has run out of available brokers to talk to (Is your cluster reachable?)' 
2019-07-05T07:39:24Z E! [telegraf] Error running agent: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

kafka -> influx

2019-07-05T07:49:34Z I! Starting Telegraf 1.11.1
2019-07-05T07:49:34Z I! Using config file: /etc/telegraf/telegraf.conf
2019-07-05T07:49:34Z I! Loaded inputs: kafka_consumer
2019-07-05T07:49:34Z I! Loaded aggregators: 
2019-07-05T07:49:34Z I! Loaded processors: 
2019-07-05T07:49:34Z I! Loaded outputs: influxdb
2019-07-05T07:49:34Z I! Tags enabled: host=436c2585696d
2019-07-05T07:49:34Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"436c2585696d", Flush Interval:10s
2019-07-05T07:49:49Z E! Error when creating Kafka Consumer, brokers: [a1:9094 a2:9094 a3:9094], topics: [test-topic]
2019-07-05T07:49:49Z E! [agent] Service for input inputs.kafka_consumer failed to start: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)
2019-07-05T07:49:49Z E! [telegraf] Error running agent: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

telnet test

$ docker exec -it test-fluent-kafka-influx-io_telegraf_k2i_1 bash -c "apt update; apt install -y telnet;telnet a1 9094"

Ign:1 http://deb.debian.org/debian stretch InRelease
Hit:2 http://security.debian.org/debian-security stretch/updates InRelease
Hit:3 http://deb.debian.org/debian stretch-updates InRelease
Hit:4 http://deb.debian.org/debian stretch Release 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
3 packages can be upgraded. Run 'apt list --upgradable' to see them.
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  telnet
0 upgraded, 1 newly installed, 0 to remove and 3 not upgraded.
Need to get 72.0 kB of archives.
After this operation, 161 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian stretch/main amd64 telnet amd64 0.17-41 [72.0 kB]
Fetched 72.0 kB in 0s (82.0 kB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package telnet.
(Reading database ... 9452 files and directories currently installed.)
Preparing to unpack .../telnet_0.17-41_amd64.deb ...
Unpacking telnet (0.17-41) ...
Setting up telnet (0.17-41) ...
update-alternatives: using /usr/bin/telnet.netkit to provide /usr/bin/telnet (telnet) in auto mode
Trying xxx.xxx.xxx.xxx...
Connected to a1.
Escape character is '^]'.

The text was updated successfully, but these errors were encountered:

phillui-37 · 2019-07-05T07:56:58Z

I don't know what is the issue root cause as I have tried to connect to same kafka service using Python->confluent-kafka which can send and receive message normally

glinton · 2019-07-05T16:08:36Z

Are you able to do a packet trace to determine where the packets are being dropped?

phillui-37 · 2019-07-08T03:18:14Z

I used tcpdump to trace packets. Result are listed here

phillui-37 · 2019-07-08T03:18:55Z

I used cloudkarafka to demo the status here

glinton · 2019-07-08T22:47:01Z

@phillui-37 do you mind trying again with the new logging addition Daniel just made?

freebsd/
- amd64
  - telegraf
  - telegraf-1.12.0~7b32fbe5_freebsd_amd64.tar.gz
- i386
  - telegraf
  - telegraf-1.12.0~7b32fbe5_freebsd_i386.tar.gz
linux/
windows/
- amd64
  - telegraf-1.12.0~7b32fbe5_windows_amd64.zip
  - telegraf.exe
- i386
  - telegraf-1.12.0~7b32fbe5_windows_i386.zip
  - telegraf.exe

phillui-37 · 2019-07-10T02:26:46Z

I used image you have provided and still have the same result

tcpdump

danielnelson · 2019-07-10T19:45:06Z

Can you run it with the --debug cli option, we are expecting to see a fair number of new log messages.

phillui-37 · 2019-07-11T02:26:26Z

log

danielnelson · 2019-07-11T03:02:30Z

Thanks @phillui-37, connection is fine but it looks like the server closes the connection once the SASL handshake is sent.

Would you be able to check the logs from one of the Kafka servers for any clues? Also, it might be helpful if you could put together a small sample program with your python code that can connect and send a message.

Finally, there is one connection option that we don't expose from Telegraf that perhaps could be of interest. Here is the library documentation:

// Whether or not to send the Kafka SASL handshake first if enabled
// (defaults to true). You should only set this to false if you're using
// a non-Kafka SASL proxy.
Handshake bool

phillui-37 · 2019-07-11T04:15:32Z

There is no logs provided by cloudkarfka free account, sorry...
well, the python test code is located here

danielnelson · 2019-07-11T04:27:50Z

Looks like support for SASL/SCRAM requires a newer version of the sarama library than we have in Telegraf right now. I'll try to put together a new build tomorrow with the updated library for testing.

IBM/sarama#1295

danielnelson · 2019-07-11T21:57:12Z

Give one of these builds a shot, you will need to add sasl_mechanism in the configurations for both the input and output:

[[outputs.kafka]]
  sasl_mechanism = "SCRAM-SHA-256"
  sasl_username = "foo"
  sasl_password = "foo"

phillui-37 · 2019-07-12T02:33:20Z

still not work

log and conf

danielnelson · 2019-07-16T00:49:01Z

I opened an issue upstream with the Sarama project: IBM/sarama#1427

marceloalmeida · 2020-04-26T19:46:02Z

Any updates? I'm still facing the same issue on version 1.14.1

vpedosyuk · 2020-07-03T15:34:46Z

It'd be great to try it with Sarama v1.26.4 as it seems to have no issues with scram-based auth

hershdhillon · 2020-09-17T21:21:49Z

Facing this today. Any updates on this particular issue?

tosheer · 2020-10-21T18:56:34Z

@danielnelson Can you share the repo of the telegraf fork which worked for you when connecting to kafka having SASL/SCRAM based authentication?

tosheer · 2020-10-21T19:08:45Z

@phillui-37 can you share telegraf fork which worked for you when connecting to kafka having SASL/SCRAM based authentication? I just need to see the implementation details.

sjwang90 · 2020-12-11T19:11:31Z

Closed in #8318

danielnelson mentioned this issue Jul 8, 2019

Emit sarama library log messages at debug level #6091

Merged

3 tasks

danielnelson mentioned this issue Jul 11, 2019

Add SCRAM SASL support to kafka plugins #6110

Closed

3 tasks

danielnelson added feature request Requests for new plugin and for new features to existing plugins upstream bug or issues that rely on dependency fixes area/kafka labels Jul 16, 2019

sjwang90 closed this as completed Dec 11, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker image cannot connect to kafka(both in/out) using SASL/SCRAM #6072

Docker image cannot connect to kafka(both in/out) using SASL/SCRAM #6072

phillui-37 commented Jul 5, 2019 •

edited

Loading

phillui-37 commented Jul 5, 2019

glinton commented Jul 5, 2019

phillui-37 commented Jul 8, 2019

phillui-37 commented Jul 8, 2019

glinton commented Jul 8, 2019 •

edited

Loading

phillui-37 commented Jul 10, 2019

danielnelson commented Jul 10, 2019

phillui-37 commented Jul 11, 2019 •

edited

Loading

danielnelson commented Jul 11, 2019 •

edited

Loading

phillui-37 commented Jul 11, 2019

danielnelson commented Jul 11, 2019

danielnelson commented Jul 11, 2019

phillui-37 commented Jul 12, 2019

danielnelson commented Jul 16, 2019

marceloalmeida commented Apr 26, 2020

vpedosyuk commented Jul 3, 2020

hershdhillon commented Sep 17, 2020

tosheer commented Oct 21, 2020

tosheer commented Oct 21, 2020

sjwang90 commented Dec 11, 2020

Docker image cannot connect to kafka(both in/out) using SASL/SCRAM #6072

Docker image cannot connect to kafka(both in/out) using SASL/SCRAM #6072

Comments

phillui-37 commented Jul 5, 2019 • edited Loading

Relevant telegraf.conf:

System info:

Steps to reproduce:

Expected behavior:

Actual behavior:

Additional info:

influx -> kafka

kafka -> influx

telnet test

phillui-37 commented Jul 5, 2019

glinton commented Jul 5, 2019

phillui-37 commented Jul 8, 2019

phillui-37 commented Jul 8, 2019

glinton commented Jul 8, 2019 • edited Loading

phillui-37 commented Jul 10, 2019

danielnelson commented Jul 10, 2019

phillui-37 commented Jul 11, 2019 • edited Loading

danielnelson commented Jul 11, 2019 • edited Loading

phillui-37 commented Jul 11, 2019

danielnelson commented Jul 11, 2019

danielnelson commented Jul 11, 2019

phillui-37 commented Jul 12, 2019

danielnelson commented Jul 16, 2019

marceloalmeida commented Apr 26, 2020

vpedosyuk commented Jul 3, 2020

hershdhillon commented Sep 17, 2020

tosheer commented Oct 21, 2020

tosheer commented Oct 21, 2020

sjwang90 commented Dec 11, 2020

phillui-37 commented Jul 5, 2019 •

edited

Loading

glinton commented Jul 8, 2019 •

edited

Loading

phillui-37 commented Jul 11, 2019 •

edited

Loading

danielnelson commented Jul 11, 2019 •

edited

Loading