[Metricbeat] Add Kafka JMX metricsets #14330

ChrsMark · 2019-10-30T14:37:25Z

Implements kafka.broker, kafka.consumer, kafka.producer metricset of #13366.

Depends on:

Manual Testing

Broker metricset

In order to prepare the proper environment to test these metricsets, the quickest way is to reuse the Docker env of the module. This can be achieved by first of all running the integration tests of the module which will take care of bringing up the container with the proper configuration.

To run the tests:
go test -v -tags=integration github.com/elastic/beats/metricbeat/module/kafka/partition

Verify that tests are successful and you can go on to manual testing. After the tests have finished the container is still alive so we can use it for the manual testing too.

Check what are the exported ports from Docker with a docker ps:

5da9631cfa25        docker.elastic.co/observability-ci/beats-integration-kafka:2.1.1-1   "/run.sh"           23 minutes ago      Up 23 minutes (healthy)   2181/tcp, 0.0.0.0:32771->8774/tcp, 0.0.0.0:32770->8775/tcp, 0.0.0.0:32769->8779/tcp, 0.0.0.0:32768->9092/tcp   metricbeat_kafka_1

In this example the mapping is:

broker port: 0.0.0.0:32769->8779
producer port: 0.0.0.0:32770->8775
consumer port: 0.0.0.0:32771->8774

So we configure the metricsets' hosts accordingly.

Enable the Kafka module and use the following configuration in modules.d/kafka.yml:

# Metrics collected from a Kafka broker using Jolokia
- module: kafka
  metricsets:
    - broker
  period: 10s
  hosts: ["localhost:32769"]

# Metrics collected from a Java Kafka consumer using Jolokia
- module: kafka
  metricsets:
    - consumer
  period: 10s
  hosts: ["localhost:32771"]

# Metrics collected from a Java Kafka producer using Jolokia
- module: kafka
  metricsets:
    - producer
  period: 10s
  hosts: ["localhost:32770"]

Run Metricbeat and verify that metrics are being collected from all of the 3 metricsets we configured.

When done remove the testing container.

Signed-off-by: chrismark <[email protected]>

jsoriano

This is looking good, I am requesting only some small changes in Dockerfile and configuration.

jsoriano · 2019-11-15T10:39:39Z

metricbeat/module/kafka/_meta/Dockerfile

@@ -8,20 +8,30 @@ ENV KAFKA_LOGS_DIR="/kafka-logs"
 ENV _JAVA_OPTIONS "-Djava.net.preferIPv4Stack=true"
 ENV TERM=linux

-RUN apt-get update && apt-get install -y curl openjdk-8-jre-headless netcat dnsutils
+RUN apt-get update && apt-get install -y curl openjdk-8-jre-headless netcat dnsutils wget


curl was already installed, you can use it, and you can also use retry as is being used to download kafka.

jsoriano · 2019-11-15T10:40:06Z

metricbeat/module/kafka/_meta/Dockerfile


 RUN mkdir -p ${KAFKA_LOGS_DIR} && mkdir -p ${KAFKA_HOME} && \
    curl -J -L -s -f -o - https://github.com/kadwanev/retry/releases/download/1.0.1/retry-1.0.1.tar.gz | tar xfz - -C /usr/local/bin && \
    retry --min 1 --max 180 -- curl -J -L -s -f --show-error -o $INSTALL_DIR/kafka.tgz \
        "https://archive.apache.org/dist/kafka/${KAFKA_VERSION}/kafka_2.11-${KAFKA_VERSION}.tgz" && \
    tar xzf ${INSTALL_DIR}/kafka.tgz -C ${KAFKA_HOME} --strip-components 1

+RUN wget -O /opt/jolokia-jvm-1.5.0-agent.jar http://search.maven.org/remotecontent\?filepath\=org/jolokia/jolokia-jvm/1.5.0/jolokia-jvm-1.5.0-agent.jar
+# RUN mv jolokia-jvm-1.5.0-agent.jar /opt/jolokia-jvm-1.5.0-agent.jar


Remove commented out code.

jsoriano · 2019-11-15T10:48:05Z

metricbeat/module/kafka/_meta/config.reference.yml

@@ -1,5 +1,5 @@
 - module: kafka
-  metricsets: ["consumergroup", "partition"]
+  metricsets: ["broker", "consumer", "consumergroup", "partition", "producer"]
  period: 10s
  hosts: ["localhost:9092"]
  enabled: true


As the hosts in the config should be different for the different metricsets, we could put it in different blocks in the reference configuration, also adding a note about the expected host, something like what we do in the kubernetes module, something like this:

# Kafka metrics collected using the Kafka protocol - module: kafka enabled: true metricsets: - consumergroup - partition period: 10s hosts: ["localhost:9092"] # Metrics collected from a Kafka broker using Jolokia #- module: kafka # enabled: true # metricsets: ['broker'] # period: 10s # hosts: ['localhost:8779'] # Metrics collected from a Java Kafka consumer using Jolokia #- module: kafka # enabled: true # metricsets: ['consumer'] # period: 10s # hosts: ['localhost:8775'] # Metrics collected from a Java Kafka producer using Jolokia #- module: kafka # enabled: true # metricsets: ['producer'] # period: 10s # hosts: ['localhost:8774']

jsoriano · 2019-11-15T11:14:52Z

metricbeat/module/kafka/_meta/config.yml

  #  - consumergroup
+  #  - producer
  period: 10s
  hosts: ["localhost:9092"]


We may need a similar separation as the one suggested above for the reference config. Or we could have just one config.yml with the suggestions mentioned before and remove the config.reference.yml.

jsoriano · 2019-11-15T11:18:16Z

metricbeat/module/kafka/_meta/run.sh

+KAFKA_OPTS="-Djava.security.auth.login.config=/kafka/bin/jaas-kafka-client-consumer.conf -javaagent:/opt/jolokia-jvm-1.5.0-agent.jar=port=8774,host=0.0.0.0" \
+ ${KAFKA_HOME}/bin/kafka-console-consumer.sh --topic=test --bootstrap-server=localhost:9091 --consumer.config ${KAFKA_HOME}/bin/sasl-producer.properties > /dev/null &
+
+wait_for_port 8774


Note to myself 🙂 consider separating it in several entrypoints for #14087.
Don't do it here because it will only complicate this PR and it is not needed by now.

jsoriano · 2019-11-15T11:21:03Z

metricbeat/module/kafka/broker/_meta/docs.asciidoc

+
+[float]
+=== Compatibility
+The module has been tested with Kafka 2.1.1. Other versions are expected to work.


We may add here also a link to the Jolokia module for the compatibility notes there.

jsoriano · 2019-11-15T11:24:51Z

metricbeat/module/kafka/broker/_meta/fields.yml

+  description: Broker metrics from Kafka Broker JMX
+  release: beta
+  fields:
+    - name: mbean


mbeans should be always in the same field at the root level (something like jmx.mbean).

This is probably something to change in the Jolokia module, so we can keep this here by now.

jsoriano · 2019-11-15T11:26:46Z

metricbeat/module/kafka/consumer/_meta/docs.asciidoc

@@ -0,0 +1,20 @@
+This metricset periodically fetches JMX metrics from Kafka Producers JMX.


We may want to explicitly mention that this only works for consumers implemented in Java, and same thing in the producer metricset.

Btw:

Suggested change

This metricset periodically fetches JMX metrics from Kafka Producers JMX.

This metricset periodically fetches JMX metrics from Kafka Consumers JMX.

jsoriano · 2019-11-15T11:29:05Z

metricbeat/module/kafka/producer/_meta/docs.asciidoc

+
+[float]
+=== Usage
+The Producer metricset requires <<metricbeat-module-jolokia,Jolokia>>to fetch JMX metrics. Refer to the link for instructions about how to use Jolokia.


We should also mention something about the metricsets requiring Jolokia in the module documentation (metricbeat/module/kafka/_meta/docs.asciidoc).

Signed-off-by: chrismark <[email protected]>

ChrsMark · 2019-11-20T16:14:11Z

Merging this one since Travis succeeded and Jenkins failure is irrelevant.

Tests will be added in a follow up PR.

(cherry picked from commit 87c49ac)

)

ChrsMark · 2020-02-06T08:46:48Z

Regression should be fixed now with the fixes at #15957.

sayden · 2020-02-06T12:41:23Z

Tested manually, good work!

ChrsMark added in progress Pull request is currently in progress. module Metricbeat Metricbeat [zube]: In Progress Team:Integrations Label for the Integrations team v7.6.0 labels Oct 30, 2019

ChrsMark requested a review from a team as a code owner October 30, 2019 14:37

ChrsMark self-assigned this Oct 30, 2019

ChrsMark mentioned this pull request Oct 30, 2019

[Metricbeat] Add New Metrics to the Kafka Module #13366

Closed

ChrsMark changed the title ~~Add kafka broker jmx~~ Add kafka jmx metrics Oct 31, 2019

ChrsMark changed the title ~~Add kafka jmx metrics~~ [Metricbeat] Add Kafka JMX metricsets Nov 1, 2019

ChrsMark added 10 commits November 12, 2019 11:12

Add kafka borker light module

12b3c58

Signed-off-by: chrismark <[email protected]>

Add docker files

cc25ce9

Signed-off-by: chrismark <[email protected]>

fix config files

84ceb2a

Signed-off-by: chrismark <[email protected]>

Add producer tasks in docker env

e9a26b0

Signed-off-by: chrismark <[email protected]>

Add producer metricset

10de02c

Signed-off-by: chrismark <[email protected]>

Add cosumer task in docker env

0f17f51

Signed-off-by: chrismark <[email protected]>

Add consumer metricset

6f24a77

Signed-off-by: chrismark <[email protected]>

Fix back run script

e6bc8db

Signed-off-by: chrismark <[email protected]>

Add fields

5ea1c60

Signed-off-by: chrismark <[email protected]>

Add docs

c2da3a3

Signed-off-by: chrismark <[email protected]>

ChrsMark force-pushed the add_kafka_broker_jmx branch from 7194cc1 to c2da3a3 Compare November 12, 2019 09:12

ChrsMark added 2 commits November 12, 2019 11:45

Move kafka jmx to oss

e98ee2f

Signed-off-by: chrismark <[email protected]>

Add go files

9426d3f

Signed-off-by: chrismark <[email protected]>

ChrsMark added the needs_backport PR is waiting to be backported to other branches. label Nov 15, 2019

ChrsMark added 3 commits November 15, 2019 10:41

Merge remote-tracking branch 'upstream/master' into add_kafka_broker_jmx

6febb14

Remove go files and update docs and list.go

7006b14

Signed-off-by: chrismark <[email protected]>

Add extra docs

3543af0

Signed-off-by: chrismark <[email protected]>

jsoriano requested changes Nov 15, 2019

View reviewed changes

ChrsMark force-pushed the add_kafka_broker_jmx branch from baba2b9 to 651fe2f Compare November 15, 2019 14:21

ChrsMark added the test-plan Add this PR to be manual test plan label Nov 20, 2019

ChrsMark added 6 commits November 20, 2019 11:27

Fix config ports

fbba3dd

Signed-off-by: chrismark <[email protected]>

Increase number of retries in healthcheck

2dfa5b8

Signed-off-by: chrismark <[email protected]>

Update docker image version

1209c62

Signed-off-by: chrismark <[email protected]>

Update x-pack configs

10f982d

Signed-off-by: chrismark <[email protected]>

Increase tests' timeout

2e5437b

Signed-off-by: chrismark <[email protected]>

Fix fields' type

54b58c2

Signed-off-by: chrismark <[email protected]>

ChrsMark merged commit 87c49ac into elastic:master Nov 20, 2019

zube bot added [zube]: Done and removed [zube]: In Progress labels Nov 20, 2019

ChrsMark added a commit to ChrsMark/beats that referenced this pull request Nov 20, 2019

[Metricbeat] Add Kafka JMX metricsets (elastic#14330)

f399d15

(cherry picked from commit 87c49ac)

ChrsMark mentioned this pull request Nov 20, 2019

Cherry-pick #14330 to 7.x: [Metricbeat] Add Kafka JMX metricsets #14656

Merged

3 tasks

ChrsMark removed the needs_backport PR is waiting to be backported to other branches. label Nov 20, 2019

ChrsMark mentioned this pull request Nov 21, 2019

Add integration/system tests for Kafka JMX metricsets #14677

Merged

ChrsMark added a commit that referenced this pull request Nov 21, 2019

Cherry-pick #14330 to 7.x: [Metricbeat] Add Kafka JMX metricsets (#14656

be8ea9e

)

andresrc removed the [zube]: Done label Nov 25, 2019

ChrsMark mentioned this pull request Nov 26, 2019

Cherry-pick #14677 to 7.x: Add integration/system tests for Kafka JMX metricsets #14784

Merged

andresrc assigned sayden and unassigned ChrsMark Jan 14, 2020

ChrsMark mentioned this pull request Jan 30, 2020

Add Kafka README for manual testing #15955

Merged

5 tasks

sayden added the test-plan-regression Manually testing this PR found a regression label Jan 30, 2020

This was referenced Jan 30, 2020

Fix packaging of OSS light modules #15957

Closed

Cherry-pick #15957 to 7.x: Fix packaging of OSS light modules #15986

Merged

Cherry-pick #15957 to 7.6: Fix packaging of OSS light modules #15987

Merged

ChrsMark mentioned this pull request Feb 4, 2020

Cherry-pick #15955 to 7.x: Add Kafka README for manual testing #16053

Merged

5 tasks

sayden added test-plan-ok This PR passed manual testing and removed test-plan-regression Manually testing this PR found a regression labels Feb 6, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Metricbeat] Add Kafka JMX metricsets #14330

[Metricbeat] Add Kafka JMX metricsets #14330

ChrsMark commented Oct 30, 2019 •

edited by zube bot

Loading

jsoriano left a comment

jsoriano Nov 15, 2019

jsoriano Nov 15, 2019

jsoriano Nov 15, 2019 •

edited

Loading

jsoriano Nov 15, 2019

jsoriano Nov 15, 2019

jsoriano Nov 15, 2019

jsoriano Nov 15, 2019

jsoriano Nov 15, 2019

jsoriano Nov 15, 2019

ChrsMark commented Nov 20, 2019

ChrsMark commented Feb 6, 2020

sayden commented Feb 6, 2020 •

edited

Loading

		@@ -0,0 +1,20 @@
		This metricset periodically fetches JMX metrics from Kafka Producers JMX.

[Metricbeat] Add Kafka JMX metricsets #14330

[Metricbeat] Add Kafka JMX metricsets #14330

Conversation

ChrsMark commented Oct 30, 2019 • edited by zube bot Loading

Manual Testing

Broker metricset

jsoriano left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsoriano Nov 15, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ChrsMark commented Nov 20, 2019

ChrsMark commented Feb 6, 2020

sayden commented Feb 6, 2020 • edited Loading

ChrsMark commented Oct 30, 2019 •

edited by zube bot

Loading

jsoriano Nov 15, 2019 •

edited

Loading

sayden commented Feb 6, 2020 •

edited

Loading