- Video
- Slides
- Overview of Architecture, Technologies & Pre-Requisites
We suggest to watch the videos in the same order as in this document.
The last video (setting up the environment) is optional, but you can check it earlier if you have troubles setting up the environment and following along the videos.
- Introduction to Docker
- Why do we need Docker
- Creating a simple "data pipeline" in Docker
- Ingesting NY Taxi Data to Postgres
- Running Posgtres locally with Docker
- Exploring the NY Taxi dataset
- Ingesting the data to the database
- Connecting pgAdmin and Postgres
- The pgAdmin tool
- Docker networks
- Putting the ingestion script to Docker
- Converting the Jupyter notebook to a Python script
- Parametrizing the script with argparse
- Dockerizing the ingestion script
- Running Postgres and pgAdmin with Docker-Compose
- Why do we need Docker-compose
- Docker-compose YAML file
- Running multiple containers with
docker-compose up
- SQL refreshser
- Adding the Zones table
- Inner joins
- Basic data quality checks
- Left, Right and Outer joins
- Group by
- Introduction to GCP (Google Cloud Platform)
- Introduction to Terraform Concepts & GCP Pre-Requisites
- Workshop: Creating GCP Infrastructure with Terraform
- Configuring terraform and GCP SDK on Windows
For the course you'll need:
- Python 3 (e.g. installed with Anaconda)
- Google Cloud SDK
- Docker with docker-compose
- Terraform
If you have problems setting up the env, you can check this video:
- Setting up the environment on cloud VM
- Generating SSH keys
- Creating a virtual machine on GCP
- Connecting to the VM with SSH
- Installing Anaconda
- Installing Docker
- Creating SSH
config
file - Accessing the remote machine with VS Code and SSH remote
- Installing docker-compose
- Installing pgcli
- Port-forwarding with VS code: connecting to pgAdmin and Jupyter from the local computer
- Installing terraform
- Using
sftp
for putting the credentials to the remote machine - Shutting down and removing the instance
- More information here
Did you take notes? You can share them here
- Notes from Alvaro Navas
- Notes from Abd
- Notes from Aaron
- Notes from Faisal
- Add your notes here