Spring Boot application used to demonstrate deployment to Kubernetes along with Kafka and Zookeeper.
This repo accompanies the following series of articles on Kafka on Kubernetes:
- Kafka on Kubernetes - Part 1: Introduction to Kubernetes: Provides an overview of Kubernetes including the key components that must be understood in order to deploy applications and expose them to external sources. Explains how minikube enables running and testing Kubernetes deployments locally, and the use of kubectl to interact with the Kubenetes cluster.
- Kafka on Kubernetes - Part 2: Deploying Kafka: Walks through deploying Kafka and Zookeeper to Kubernetes, and explains the kubectl commands used to query the state of the deployment. Steps through sending and receiving events to the deployed Kafka instance using the Kafka command line tools.
- Kafka on Kubernetes - Part 3: Spring Boot Demo: Walks through deploying a Spring Boot application to Kubernetes. The application connects to the deployed Kafka to consume and produce events. It provides a REST API enabling a client to trigger sending events to Kafka, and the steps to expose this to an external source are described.
The application provides a REST endpoint that accepts a request to trigger sending events. The number of events to produce can be specified.
The application also consumes an event that triggers sending outbound events. The number of events to produce can be specified.
Kafka, Zookeeper, and the demo Spring Boot application are deployed as Docker containers to minikube, which is an implementation of Kubernetes that is great for local testing. Interacting with the application and Kafka are demonstrated by calling the REST endpoint to trigger emitting events, and by sending events to Kafka and receiving the emitted events from the application via the Kafka commandline tools.
Select the version of minikube suitable for the OS following step 1 here:
https://minikube.sigs.k8s.io/docs/start/
e.g. for macOS ARM64:
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-darwin-arm64
sudo install minikube-darwin-arm64 /usr/local/bin/minikube
First start Docker, then start the cluster:
minikube start
View cluster state in the browser:
minikube dashboard
At this point there is nothing deployed to view.
Note that minikube can be stopped with the following:
minikube stop
A namespace can be used rather than using the default one to isolate the resources that are created.
A Kubernetes namespace template is provided in the /resources directory in the root of the project. To create the namespace called demo run:
kubectl create -f ./resources/demo-namespace.yml
The response confirms the namespace is created:
namespace/demo created
View namespaces:
kubectl get namespaces
Individual commands can now be directed at the demo namespace by adding the option --namespace=demo. Alternatively the namespace can be configured as the default by running:
kubectl config set-context $(kubectl config current-context) --namespace=demo
To delete the namespace, run:
kubectl delete namespace demo
Kubernetes deployment manifests are provided for Kafka and Zookeeper in the /resources directory. To create the pods for Kafka and Zookeeper run:
kubectl create -f ./resources/zookeeper.yml
kubectl create -f ./resources/kafka.yml
Kafka and Zookeeper pods can now be viewed in the minikube dashboard. Alternatively from the command line:
kubectl get pods
This provides the pod names which can be used for later commands.
To view the deployments:
kubectl get deployments
To view the services:
kubectl get services
View the logs (with the pod name obtained via kubectrl get pod):
kubectl logs kafka-deployment-57f8cd77f6-v2gts
To delete all pods:
kubectl delete --all pods
Note that as the pods are managed by the deployment, as the pod counts drop below the required replica count, they are recreated.
To permanently delete the pods, the deployment must be deleted:
kubectl delete --all deployments
The Spring Boot demo application uses two topics, demo-inbound-topic and demo-outbound-topic. These should be created in advance.
First jump onto the Kafka pod:
kubectl exec -it kafka-deployment-57f8cd77f6-v2gts -- /bin/bash
Then execute the following commands to create the topics:
kafka-topics --bootstrap-server localhost:9092 --create --topic demo-inbound-topic --replication-factor 1 --partitions 3
kafka-topics --bootstrap-server localhost:9092 --create --topic demo-outbound-topic --replication-factor 1 --partitions 3
The response should show the topics created:
Created topic demo-inbound-topic.
Created topic demo-outbound-topic.
Ensure the bootstrap-server URL in the Spring Boot application properties, located in src/main/resources/application.yml, points to the Kafka service:
kafka:
bootstrap-servers: kafka-service:9092
The deployment manifest for the Spring Boot application is located in resources/demo-application.yml.
The Spring Boot application is built using maven:
mvn clean install
Kubernetes uses its own local Docker registry that is not connected to the Docker registry on the local machine. As the Spring Boot application Docker image will be built locally and is not available in the public Docker registry, the imagePullPolicy is set to Never in the resources/demo-application.yml. The following minikube command is then required in order to output the environment variables that are needed to point the local Docker daemon to the minikube internal Docker registry.
eval $(minikube -p minikube docker-env)
This command must be run in any new terminal window opened to take effect for the next Docker command.
Then the Docker image can be built for the application. This will installed into the minikube registry, ready for deployment to Kubernetes:
docker build -t kube/kafka-kubernetes-demo:latest .
Finally the Spring Boot pod can be created and deployed, and is viewable in the minikube dashboard:
kubectl create -f ./resources/demo-application.yml
Console output shows the deployment and service are created:
deployment.apps/kafka-kubernetes-demo created
service/kafka-kubernetes-demo-service created
kafka-kubernetes-demo-service is the name defined in the service metadata name from the ./resources/demo-application.yml:
The job can be removed and recreated if necessary, by first deleting it:
kubectl delete -f ./resources/demo-application.yml
Assign an external port by starting a tunnel (the namespace will be required here):
minikube service kafka-kubernetes-demo-service --namespace demo
This will open the root page of the Spring Boot application in the browser, which displays Spring Boot Demo, verifying the application has started successfully.
Alternatively to get the URL without opening the browser, use the --url option:
minikube service kafka-kubernetes-demo-service --namespace demo --url
With the tunnel open, hit the version endpoint on the Spring Boot application using curl.
curl -X GET http://localhost:53298/v1/demo/version
This should return v1
Send a REST request to trigger the application into sending events to Kafka:
curl -v -d '{"numberOfEvents":3}' -H "Content-Type: application/json" http://localhost:53298/v1/demo/trigger
Check Spring Boot application logs to confirm events have been sent:
kubectl logs kafka-kubernetes-demo-745d47966c-tmvqq
13:23:34.320 INFO d.k.KafkaDemoApplication - Started KafkaDemoApplication in 1.423 seconds (process running for 1.648)
14:14:43.734 INFO d.k.s.DemoService - Sending 3 events
14:14:45.318 INFO d.k.s.DemoService - Total events sent: 3
Jump onto the Kafka pod:
kubectl exec -it kafka-deployment-57f8cd77f6-v2gts -- /bin/bash
Consume the events already produced from the beginning of the demo-outbound-topic:
kafka-console-consumer --bootstrap-server localhost:9092 --topic demo-outbound-topic --from-beginning
The events, populated with randomised names, are output:
{"firstName":"ZrdOwISesV","middleName":"FXjMEVAyZf","lastName":"WRfyXxHSGB"}
{"firstName":"qTZORIeXpU","middleName":"qCurtqohni","lastName":"mWUIEeTSQk"}
{"firstName":"xvlSzBpKVZ","middleName":"HjNQjirBcu","lastName":"MwzFClWjHV"}
Trigger sending new events from the application by submitting a request to the demo-inbound-topic, and observe the new events being emitted. Use a new terminal window in order to leave the console-consumer running.
kafka-console-producer --broker-list localhost:9092 --topic demo-inbound-topic
{"numberOfEvents":2}
This should be reflected in the logs:
14:25:09.789 INFO d.k.c.KafkaDemoConsumer - Received message - event: DemoInboundEvent(numberOfEvents=2)
14:25:09.789 INFO d.k.s.DemoService - Sending 2 events
14:25:09.818 INFO d.k.s.DemoService - Total events sent: 2
https://kubernetes.io/docs/tasks/access-application-cluster/ingress-minikube/
minikube addons enable ingress
Verify running with:
kubectl get pods -n ingress-nginx
Create the ingress for the kafka-kubernetes-demo-service:
kubectl apply -f ./resources/demo-ingress.yml
The output on the console should show this as created:
ingress.networking.k8s.io/demo-ingress created
The ingress has been created with host k8s.springboot.demo. To view the ingress:
kubectl get ingress
The ingress can be deleted with the following:
kubectl delete ingress demo-ingress
Update the /etc/hosts file, adding the following line (correct for MacOS ARM):
127.0.0.1 k8s.springboot.demo
For other operating systems add the minikube IP to the /etc/hosts file instead:
192.168.49.2 k8s.springboot.demo
Now start the minikube tunnel:
minikube tunnel
Open the following address in the browser to see the landing page:
http://k8s.springboot.demo
Alternatively verify by getting the application version using curl:
curl -X GET http://k8s.springboot.demo/v1/demo/version
This returns the String v1.