ullala

ullala is a demo that showcase using kafka as a data pipeline and log aggregation through the ELK stack

Setting up

1. Download kafka command line tools: scripts/download-kafka.sh
2. Run Kafka, Zookeeper, Elasticsearch and Logstash: docker-compose up
3. Set kafka on your host file to point to the running kafka instance (docker-machine ip default on mac)
4. Setup topics scripts/setup-topics.sh
5. Install OpenCV for capturer, brew install opencv on mac.
6. Symlink opencv into capturer, for mac you can run scripts/symlink-opencv-bew.sh
7. Install capturer dependencies pip install -r capturer/requirements.txt
8. Run Capturer: KAFKA_HOSTS=KAFKA:9092 capturer/capturer.py
9. Install processor dependencies cd processor && npm insall
10. Run processor: cd processor && KAFKA_HOSTS=kafka:9092 node index.js
11. Install pusher dependencies cd pusher && npm install
12. Run pusher `cd pusher && KAFKA_HOSTS=kafka:9092 node index.js