airflow-on-EKS

This is an implementation of airflow on AWS EKS with official Helm chart.

AWS EFS is used as dags/logs persistent volume. You can setup a EFS provisioner with this guide. https://github.com/kubernetes-incubator/external-storage/tree/master/aws/efs

Customized git-sync sidecar is used to deploy Airflow Dags from remote repo to Airflow server for the following reasons.

Not all files under the DAGs repo need to be deployed to Airflow server.
If files under dags folder in Airflow server are changed, git-sync may not be able to pull from remote repo.

K8s deployment

EFS and storage provisioner https://github.com/kubernetes-sigs/aws-efs-csi-driver
Reloader to refresh changes on config maps https://github.com/stakater/Reloader
Flux with helm operator (v1) https://fluxcd.io/legacy/helm-operator/
kubernetes external secret and AWS secret manager https://github.com/external-secrets/kubernetes-external-secrets

kubectl apply -f k8s_config/

Copy .local.env to .env and update the values in it.
Run for the first time.
```
docker volume create --name=airflow_db
```
Run
```
docker compose up -d
```
Wait for webserver and scheduler to be ready (1st time may take longer and may require restart)
Go to http://localhost:8080

username: airflow

password: airflow