kafka-connect

Sample showing how to use Marten together with Docker and Kafka Connect.

See more about Marten in documentation: link.

The aim of this example is to show how to model following flow:

1. Marten is .NET framework that allows to store data in Posgres as Documents and Events. Document and Event Data is stored with document tables.

2. Data from the Marten tables will be moved through Debezium framework with Postgres CDC mechanism (sink connector).

   2.1 Events table (mt_events) with Outbox pattern. See more here and there.

   2.2 Document tables with "regular" approach.

3. Debezium moves data to Kafka Connect and then to Kafka.

4. Documents (mt_doc_meetings) are moved then to ElasticSearch through Kafka Connect sink connector.

How to run sample?

1. Run following docker commands:

   1.1. docker-compose build - to build image of the .NET sample app,

   1.2. docker-compose up - to start containers.

2. Docker compose file has all needed setup to do the work:

   2.1. Postgres with Debezium preconfigured - see config

   2.1. Kafka-Conect - see config

   2.2. ElasticSearch - see config

   2.3. Zookeeper - see config

   2.4. Kafka - see config

   2.5. Avro Schema Registry - see config

   2.6. Kafka Rest - see config

   2.7. Sample .NET WebApi app with Marten - config.

3. Besides the tools by themselves you can also use UIs:

   3.1. PGAdmin - http://localhost:5050 - see config

   3.2. Kafka Topics UI - http://localhost:8000 - see config

   3.3. Kibana - UI for ElasticSearch - http://localhost:5601 - see config

   3.4. Swagger for Marten WebApi - http://localhost:5000/swagger/index.html

4. Started dockers already have test tables preconfigured - see SQL script.

5. Connectors are configured via CURL - see Script and run via docker compose.

Links

• https://medium.com/trendyol-tech/debezium-with-simple-message-transformation-smt-4f5a80c85358

Kafka Connect with Marten is Copyright © 2018-2021 Oskar Dudycz and other contributors under the MIT license.