Run some example in Kubernetes execution mode in CI

[pankajastro]

Description

Migrate example from cosmos-example

The cosmos-example repository currently contains several examples, including those that run in Kubernetes execution mode. This setup has made testing local changes in Kubernetes execution mode challenging and keeping the documentation up-to-date is also not easy. Therefore, it makes sense to migrate the Kubernetes examples from cosmos-example to this repository. This PR resolved the below issue in this regard

• Migrate the jaffle_shop_kubernetes example DAG to the this repository.
• Moved the Dockerfile from cosmos-example to this repository to build the image with the necessary DAGs and DBT projects
  I also adjusted both the example DAG and Dockerfile to work within this repository.

Automate running locally

I introduce some scripts to make running Kubernetes DAG easy.

postgres-deployment.yaml: Kubernetes resource file for spinning up PostgreSQL and creating Kubernetes secrets.

integration-kubernetes.sh: Runs the Kubernetes DAG using pytest.

kubernetes-setup.sh:

• Builds the Docker image with the Jaffle Shop dbt project and DAG, and loads the Docker image into the local registry.
• Creates Kubernetes resources such as PostgreSQL deployment, service, and secret.

Run DAG locally
Prerequisites:

• Docker Desktop
• KinD (Kubernetes in Docker)
• kubectl

Steps:

1. Create cluster: kind create cluster
2. Create Resource: scripts/test/kubernetes-setup.sh (This will set up PostgreSQL and load the DBT project into the local registry)
3. Run DAG: cd dev && scripts/test/integration-kubernetes.sh this will execute this DAG with a pytest you can also run directly with airflow command given that project is installed in your virtual env

time AIRFLOW__COSMOS__PROPAGATE_LOGS=0 AIRFLOW__COSMOS__ENABLE_CACHE=1 AIRFLOW__COSMOS__CACHE_DIR=/tmp/ AIRFLOW_CONN_EXAMPLE_
CONN="postgres://postgres:[email protected]:5432/postgres" PYTHONPATH=`pwd` AIRFLOW_HOME=`pwd` AIRFLOW__CORE__DAGBAG_IMPORT_TIMEOUT=20000 AIRFLOW__CORE__DAG_FILE_PROCESSOR_TIMEOUT=20000 airflow dags test jaffle_shop_kubernetes  `date -Iseconds`

Run jaffle_shop_kubernetes in CI

To avoid regression we have automated running the jaffle_shop_kubernetes in CI

• Set up the GitHub Actions infrastructure to run DAGs using Kubernetes execution mode
• Use container-tools/kind-action@v1 to create a KinD cluster.
• Used the bash script to streamline the creation of Kubernetes resources, build and load the image into a local registry, and execute tests.
• At the moment I'm running the pytest from virtual env

Documentation changes

Given that the DAG jaffle_shop_kubernetes is now part of this repository, I have automated the example rendering for Kubernetes execution mode. This ensures that we avoid displaying outdated example code.

https://astronomer.github.io/astronomer-cosmos/getting_started/execution-modes.html#kubernetes

https://astronomer.github.io/astronomer-cosmos/getting_started/kubernetes.html#kubernetes

Future work

• Use the hatch target to run the test. I have introduced the hatch target to run the Kubernetes example with hatch, but it's currently not working due to a mismatch between the local and container DBT project paths. This requires a bit more work.
• Remove the virtual environment step (Install packages and dependencies) in the CI configuration for Run-Kubernetes-Tests and use hatch instead.
• Update the profile YAML to use environment variables for the port, as it is currently hardcoded.
• Remove the host from the Kubernetes secret and replace it with the username and make corresponding change in DAG
• Currently, we need to export both POSTGRES_DATABASE and POSTGRES_DB in the Dockerfile because both are used in the project. To ensure consistency, avoid exporting both and instead make the environment variables consistent across the repository
• Not a big deal in this context, but we have some hardcoded values for secrets. It would be better to parameterize them

GH issue for future improvement: #1160

Example CI Run

• https://github.com/astronomer/astronomer-cosmos/actions/runs/10405590862

Related Issue(s)

closes: #535

Breaking Change?

No

Checklist

• I have made corresponding changes to the documentation (if required)
• I have added tests that prove my fix is effective or that my feature works

[netlify]

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name	Link
🔨 Latest commit	05cedca
🔍 Latest deploy log	https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/66be350363b2af0007e7a6e4

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.37%. Comparing base (a89389d) to head (05cedca).
Report is 3 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1127   +/-   ##
=======================================
  Coverage   96.37%   96.37%           
=======================================
  Files          64       64           
  Lines        3424     3424           
=======================================
  Hits         3300     3300           
  Misses        124      124           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[tatiana]

This looks great, @pankajastro , really happy this is automated, it will save us lots of time!