Plan

In this article we will setup a simple kafka cluster with 2 nodes in it.

We will validate that cluster is working by producing and consuming messages. We will setup kafka-ui to see the cluster details. We will also export cluster metrics and watch them in grafana.

We will use docker compose for this to make things simpler.

The docker image for the kafka will be bitnami/kafka/.

Kafka cluster

Let's start with this docker-compose.yml:

version: '3.8'

x-kafka-common: &kafka-common
  image: 'bitnami/kafka:latest'
  ports:
    - "9092"
  networks:
    - kafka
  healthcheck:
    test: "bash -c 'printf \"\" > /dev/tcp/127.0.0.1/9092; exit $$?;'"
    interval: 5s
    timeout: 10s
    retries: 3
    start_period: 30s
  restart: unless-stopped

x-kafka-env-common: &kafka-env-common
  ALLOW_PLAINTEXT_LISTENER: 'yes'
  KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE: 'true'
  KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: 0@kafka-0:9093,1@kafka-1:9093
  KAFKA_KRAFT_CLUSTER_ID: abcdefghijklmnopqrstuv
  KAFKA_CFG_PROCESS_ROLES: controller,broker
  KAFKA_CFG_CONTROLLER_LISTENER_NAMES: CONTROLLER
  KAFKA_CFG_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
  EXTRA_ARGS: "-Xms128m -Xmx256m"

services:

  kafka-0:
    <<: *kafka-common
    environment:
      <<: *kafka-env-common
      KAFKA_CFG_NODE_ID: 0
    volumes:
      - kafka_0_data:/bitnami/kafka

  kafka-1:
    <<: *kafka-common
    environment:
      <<: *kafka-env-common
      KAFKA_CFG_NODE_ID: 1
    volumes:
      - kafka_1_data:/bitnami/kafka

networks:
  kafka:

volumes:
  kafka_0_data:
    driver: local
  kafka_1_data:
    driver: local

Here we define 2 services: kafka-0 and kafka-1.

We don't need any encryption, so we use ALLOW_PLAINTEXT_LISTENER=yes. Also we don't use zookeeper, instead we will use KRaft protocol.

We will add a healthcheck command, which will later help us to run dependent services. The reason why we do not use curl is because the response will be empty, and it will make curl produce an error

curl: (52) Empty reply from server

As a result, node will be marked as unhealthy.

On each node we need to set its id with this variable: KAFKA_CFG_NODE_ID.

By setting KAFKA_CFG_CONTROLLER_QUORUM_VOTERS environment variable on both services, we make them to work in a cluster. Also we will use random value for the cluster id with the variable KAFKA_KRAFT_CLUSTER_ID: just use any string.

We set also KAFKA_CFG_PROCESS_ROLES, KAFKA_CFG_CONTROLLER_LISTENER_NAMES and KAFKA_CFG_LISTENERS because they do not have default values.

All other variables will use their default values, you can read more about them here: https://github.com/bitnami/containers/blob/main/bitnami/kafka/README.md#configuration.

Let's run our cluster:

docker compose up

Validate cluster

Now we need to create a new topic and put some messages into it. Kafka comes with a set of scripts for managing it, let's use them:

docker compose exec kafka-0 /opt/bitnami/kafka/bin/kafka-topics.sh --create \
    --bootstrap-server kafka-0:9092,kafka-1:9092 --replication-factor 1 --partitions 1 --topic test

We tell the kafka-topics.sh that it should use our kafka instances as a bootstrap servers. As a result you should see

Created topic test.

Let's list all topics:

$ docker compose exec kafka-0 /opt/bitnami/kafka/bin/kafka-topics.sh --list --bootstrap-server kafka-0:9092,kafka-1:9092
test
$ docker compose exec kafka-0 /opt/bitnami/kafka/bin/kafka-topics.sh --describe --topic test --bootstrap-server kafka-0:9092,kafka-1:9092
Topic: test	TopicId: 7FEciEQRRjqJ2zntIONOcQ	PartitionCount: 1	ReplicationFactor: 1	Configs: segment.bytes=1073741824
	Topic: test	Partition: 0	Leader: 0	Replicas: 0	Isr: 0

Let's write some messages into this topic. Open new terminal (this will be producer terminal) and type:

docker compose exec kafka-0 /opt/bitnami/kafka/bin/kafka-console-producer.sh --bootstrap-server kafka-0:9092,kafka-1:9092 --producer.config /opt/bitnami/kafka/config/producer.properties --topic test
>message0

Open another terminal (consumer terminal) and read the message with a consumer:

docker compose exec kafka-0  /opt/bitnami/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server kafka-0:9092,kafka-1:9092 --consumer.config /opt/bitnami/kafka/config/consumer.properties --topic test --from-beginning

You should see message0. Switch back to the producer terminal and type another message. You will see it immediately in consumer terminal.

Press Control+C to close producer and consumer.

At this point we can start a new cluster, create topic, produce the messages and consume them.

Kafka UI

Next we will add kafka-ui. This application will give us a nice UI to view our cluster.

Add the following to the docker-compose.yml:

  kafka-ui:
    container_name: kafka-ui
    image: provectuslabs/kafka-ui:latest
    volumes:
      - ./kafka-ui/config.yml:/etc/kafkaui/dynamic_config.yaml
    environment:
      DYNAMIC_CONFIG_ENABLED: 'true'
    depends_on:
      - kafka-0
      - kafka-1
    networks:
      - kafka
    ports:
      - '8080:8080'
    healthcheck:
      test: wget --no-verbose --tries=1 --spider localhost:8080 || exit 1
      interval: 5s
      timeout: 10s
      retries: 3
      start_period: 30s

You need to create a configuration file for the kafka ui:

mkdir -p kafka-ui
touch kafka-ui/config.yml

Kafka ui can be configured with environment variables, but I prefer yaml, because variables will become very messy once you will have more then 1 cluster.

Put the following into kafka-ui/config.yml:

auth:
  type: LOGIN_FORM

spring:
  security:
    user:
      name: admin
      password: admin

kafka:
  clusters:
    - bootstrapServers: kafka-0:9092,kafka-1:9092
      name: kafka

Stop current docker compose process (control+c) and restart it:

docker compose up

Open kafka ui in the browser http://localhost:8080. Use admin:admin as a credentials (see kafka-ui/config.yml). You will see our new cluster.

Visit http://localhost:8080/ui/clusters/kafka/all-topics/test/messages?keySerde=String&valueSerde=String&limit=100 and make sure you see all messages, that we produced earlier.

Metrics

Kafka exporter

Now we need to expose kafka metrics to the prometheus and be able to see them in grafana. Kafka exporter will be used to actually export the metrics. Let's add new services to the docker-compose.yml:

  prometheus:
    image: prom/prometheus
    container_name: prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
    ports:
      - 9090:9090
    volumes:
      - ./prometheus:/etc/prometheus
      - prom_data:/prometheus
    networks:
      - kafka  
    healthcheck:
      test: wget --no-verbose --tries=1 --spider localhost:9090 || exit 1
      interval: 5s
      timeout: 10s
      retries: 3
      start_period: 5s
      
  kafka-exporter:
    image: docker.io/bitnami/kafka-exporter:latest
    depends_on:
      kafka-0:
        condition: service_healthy
      kafka-1:
        condition: service_healthy
    networks:
      - kafka
    command: --kafka.server=kafka-0:9092 --kafka.server=kafka-1:9092
    healthcheck:
      test: "bash -c 'printf \"\" > /dev/tcp/127.0.0.1/9308; exit $$?;'"
      interval: 5s
      timeout: 10s
      retries: 3
      start_period: 5s
      
        
  grafana:
    image: grafana/grafana
    container_name: grafana
    ports:
      - 3000:3000
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=grafana
    volumes:
      - ./grafana/provisioning:/etc/grafana/provisioning
      - ./grafana/dashboards:/var/lib/grafana/dashboards
    networks:
      - kafka
    healthcheck:
      test: curl --fail localhost:3000
      interval: 5s
      timeout: 10s
      retries: 3
      start_period: 10s

and this in volumes section:

  prom_data:
    driver: local

Kafka exporter depends on kafka instances, it will produce an error during start, if there are no running kafka brokers. This is why we added the healthcheck for kafka.

JMX exporter

There is also one more way to collect metrics (they can be used together): jmx exporter. We will use it as a java agent.

Download the jar and the config:

mkdir -p jmx-exporter
curl https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.19.0/jmx_prometheus_javaagent-0.19.0.jar \
  -o jmx-exporter/jmx_prometheus_javaagent-0.19.0.jar
curl  https://raw.githubusercontent.com/prometheus/jmx_exporter/main/example_configs/kafka-2_0_0.yml \
  -o jmx-exporter/kafka-2_0_0.yml

Mount it to kafka images:

  kafka-0:
    <<: *kafka-common
    environment:
      <<: *kafka-env-common
      KAFKA_CFG_NODE_ID: 0
    volumes:
      - kafka_0_data:/bitnami/kafka
      - ./jmx-exporter:/opt/jmx-exporter

  kafka-1:
    <<: *kafka-common
    environment:
      <<: *kafka-env-common
      KAFKA_CFG_NODE_ID: 1
    volumes:
      - kafka_1_data:/bitnami/kafka
      - ./jmx-exporter:/opt/jmx-exporter

Update the x-kafka-env-common block in docker-compose.yml like this:

x-kafka-env-common: &kafka-env-common
  ALLOW_PLAINTEXT_LISTENER: 'yes'
  KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE: 'true'
  KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: 0@kafka-0:9093,1@kafka-1:9093
  KAFKA_KRAFT_CLUSTER_ID: abcdefghijklmnopqrstuv
  KAFKA_CFG_PROCESS_ROLES: controller,broker
  KAFKA_CFG_CONTROLLER_LISTENER_NAMES: CONTROLLER
  KAFKA_CFG_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
  EXTRA_ARGS: "-Xms128m -Xmx256m -javaagent:/opt/jmx-exporter/jmx_prometheus_javaagent-0.19.0.jar=9404:/opt/jmx-exporter/kafka-2_0_0.yml"

The exporter will be available at port 9404:

docker compose exec kafka-0 curl localhost:9404

We also want Kafka UI to use the metrics. Add this to kafka-ui/config.yml:

kafka:
  clusters:
    - bootstrapServers: kafka-0:9092,kafka-1:9092
      name: kafka
      metrics:
        type: JMX
        port: 9404

Prometheus

Create configuration file for a prometheus:

mkdir -p prometheus
touch prometheus/prometheus.yml

Put the following content:

global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 15s
scrape_configs:
- job_name: kafka-exporter
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  static_configs:
  - targets:
    - kafka-exporter:9308
- job_name: jmx-exporter
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  static_configs:
  - targets:
    - kafka-0:9404
    - kafka-1:9404

Here we ask prometheus to get metrics from kafka exporter from /metrics endpoint on port 9093 and from jmx-exporter on port 9404.

Grafana

Finally, create the configuration for grafana:

mkdir -p grafana/provisioning/datasources
mkdir -p grafana/provisioning/dashboards
touch grafana/provisioning/datasources/datasource.yml
touch grafana/provisioning/dashboards/dashboard.yml
curl https://raw.githubusercontent.com/strimzi/strimzi-kafka-operator/main/examples/metrics/grafana-dashboards/strimzi-kafka-exporter.json -o grafana/dashboards/strimzi-kafka-exporter.json
curl https://raw.githubusercontent.com/strimzi/strimzi-kafka-operator/main/examples/metrics/grafana-dashboards/strimzi-kafka.json -o grafana/dashboards/strimzi-kafka.json

We tell the grafana to provision prometheus datasource. Also we download the dashboard for the kafka exporter and jmx exporter and tell grafana to use it.

Put the following into grafana/provisioning/datasources/datasource.yml:

apiVersion: 1

deleteDatasources:
  - name: Prometheus
    orgId: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    orgId: 1
    url: http://prometheus:9090
    password:
    user:
    database:
    basicAuth: false
    basicAuthUser:
    basicAuthPassword:
    withCredentials:
    isDefault: true
    version: 1
    editable: false

and the following into grafana/provisioning/dashboards/dashboard.yml:

apiVersion: 1

providers:
  - name: 'dashboards-from-file'
    orgId: 1
    folder: ''
    folderUid: ''
    type: file
    disableDeletion: false
    updateIntervalSeconds: 10
    allowUiUpdates: false
    options:
      path: /var/lib/grafana/dashboards
      foldersFromFilesStructure: true

Now restart docker compose and visit http://localhost:3000/dashboards. Use admin:grafana as credentials. Open dashboard with name Strimzi Kafka Exporter, you will see some details about the cluster.

Conclusion

Now we can run grafana cluster, we can validate it by producing and consuming messages from CLI. Also we can view the details of the cluster with UI and watch the cluster metrics in grafana.

Links

APACHE KAFKA: https://kafka.apache.org/
Bitnami kafka image: https://hub.docker.com/r/bitnami/kafka/
Grafana: https://grafana.com/
jmx_exporter: https://github.com/prometheus/jmx_exporter
Kafka Exporter: https://github.com/danielqsj/kafka_exporter
Kafka UI: https://github.com/provectus/kafka-ui
Prometheus: https://prometheus.io/

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Plan

Kafka cluster

Validate cluster

Kafka UI

Metrics

Kafka exporter

JMX exporter

Prometheus

Grafana

Conclusion

Links

About

Releases

Packages

Name		Name	Last commit message	Last commit date
Latest commit History 6 Commits
grafana/provisioning		grafana/provisioning
kafka-ui		kafka-ui
prometheus		prometheus
.gitignore		.gitignore
README.md		README.md
docker-compose.yml		docker-compose.yml

papirosko/kafka-demo

Folders and files

Latest commit

History

Repository files navigation

Plan

Kafka cluster

Validate cluster

Kafka UI

Metrics

Kafka exporter

JMX exporter

Prometheus

Grafana

Conclusion

Links

About

Resources

Stars

Watchers

Forks

Releases

Packages 0

Packages