A small example on how to use Kafka in AWS MSK.
For detailed instructions, refer to the original Confluence page. Below are the steps to create and run a local Docker image for Kafka-UI:
Note: Adapt the following code to dev
or prod
accordingly.
-
Build the Docker image using the
kafka-ui-dockerfile.txt
:docker build -t kafkaui:local . -f kafka-ui-dockerfile.txt
-
Run the Docker container manually or using Docker Desktop:
docker run -p 8080:8080 -d kafkaui:local
Creating a topic in Kafka-UI is straightforward. For simple topics, assign a name and the number of partitions. By default, we use 9 partitions.
-
Create a
poetry.toml
file with the following content:[virtualenvs] in-project = true
-
Initialize the Python environment:
poetry init --python ">=3.10.6,<3.11"
-
Add Cloudsmith to access Aily packages:
poetry source add cloudsmith https://dl.cloudsmith.io/PRBlnvv3GHFM3KDl/ailylabs/pip/python/simple/
-
Add the required libraries:
poetry add aily-py-commons poetry add aily-ai-brain poetry add jupyter poetry add confluent-kafka
There are two notebooks: one for the consumer and one for the producer. Execute the consumer notebook, followed by the producer notebook.
- The consumer will print any received messages.
- The producer will send a simple "Hello Kafka" message.