Kafka Labs

Experiments with kafka and the kafka confluent platform running on top of kubernetes.

Requirements

• asdf version manager
• helm and terraform plugins for asdf
• Docker
• k3d dockerized k3s helper

Before beginning with the setup run the following command:

asdf install

Setup

To setup a local run the following command:

make start

This will start the following components in a local k8s single node cluster:

• Kafka
• ZooKeeper
• Kafka-Connect
• MSSQL

To configure the kubeconfig in your terminal:

export KUBECONFIG=$(k3d get-kubeconfig --name=kafka-labs)

Once your done stop the cluster:

make stop

Example

Deploy Connectors

Make sure to forward the ports of the services:

make pf

In another terminal enter the following commands to install the debezium mssql plugin:

cd terraform
terraform apply

Insert Data to the MSSQL Instance

Make sure to forward the ports of the services:

make pf

In another terminal enter the following command:

make producer

This will insert 3000 rows into the ship and train tables every second for the next hour.

Listening to a kafka topic

Make sure to forward the ports of the services:

make pf

In another terminal enter the following command:

make listen TOPIC=main.dbo.ship

Monitoring

To monitor the pod that runs kafka connect:

make monitoring