Apache Kafka and Apache Spark - A Distributed Streaming Project

For detailed project report, please check out The Project Report. ;)

Running the project

You can try out this project running all components in a docker composed cluster, each component will run in a separated container and they will be connected with the default network inside the compose cluster.

Prerequisites:

• Docker
• Port 80 should be available or change the port mapping of the frontend app in docker-compose.yml

Run the project:

• Clone this repository on your computer
• Fire up the docker compose cluster:

$ docker-compose up

• Open the frontend application in your browser: $ open http://localhost:80

Shutting down docker-compose (use an other terminal window for running this command):

$ docker-compose down

Notes:

• Building the maven based projects at the first time takes a while, please be patient.
• Kafka generated files are mapped to ./kafka/volumes folder. If the streaming does not start when the project is launched at first time, shut down the docker-compose cluster and start it up again. All mapped volumes and folders should be available at the second launch and Kafka can start properly.

Run the project in developer mode

You can run all components locally. In this way you can easily debug and add new features.

Prerequisites:

• Locally installed Node.js (How to Install Node.js)
• Locally installed Java 8 and Maven 3.5
• Locally installed Kafka with Zookeeper (Kafka Quickstart)
• Locally installed Spark (Install Spark)

The setup npm script will install individual packages and prepare the project for you. The start:dev will run all component's development script concurrently in the same terminal.

$ npm run setup
$ npm run start:dev

Documents

• Final Report
• Original project proposal
• Notes about datasources

Component's README files

• Backend
• API framework, Nest.js original README
• Frontend
• Frontend framework, React.js original README
• Kafka
• SparkStreamer

Useful links

• Awesome Streaming
• Exactly-once Support in Apache Kafka
• Understanding When to use RabbitMQ or Apache Kafka
• Why need Zookeeper for Kafka?

Reading

• 7 essential technologies for a modern data architecture
• 5 Layered architecture