Airflow Kubernetes Executor Mounting Volumes to Worker Pods

[ngreen-ny]

Good day, we are looking at switching from the LocalExecutor to the KubernetesExecutor in Airflow. I have one question though.

I have created a docker image with airflow to execute my task pods on it.

The only issue is that the dag volumes don't get mounted to the worker pod so the dag cannot be found. The pod template spec has the correct volumes showing.

Airflow Version v2.2.3

Here is part of my configmap for the pod template spec:

      volumeMounts:
        - name: dags-data
          mountPath: /dags
          subPath:
          readOnly: true
        - name: logs-data
          mountPath: /opt/airflow/logs
          subPath:
  volumes:
    - name: dags-data
      persistentVolumeClaim:
        claimName: airflow-test-dags
    - name: logs-data
      persistentVolumeClaim:
        claimName: airflow-test-logs

---

When the task pod runs the volumes are not mounted:

Volumes:
default-token-kxpnv:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-kxpnv
Optional: false
QoS Class: BestEffort

Ideally I would like to have these volumes by default and not have to define them in each dag. Is that possible to get them mounted to the worker pod. Here are my config below:

AIRFLOW__KUBERNETES__DAGS_IN_IMAGE: "False"
AIRFLOW__KUBERNETES__DELETE_WORKER_PODS: "True"
AIRFLOW__KUBERNETES__WORKER_CONTAINER_REPOSITORY: "myinternalrepo.com/airflow-dev-k8s-worker"
AIRFLOW__KUBERNETES__WORKER_CONTAINER_TAG: "0.0.1"
AIRFLOW__KUBERNETES__RUN_AS_USER: "50000"
AIRFLOW__KUBERNETES__DAGS_VOLUME_CLAIM: "airflow-test-dags"
AIRFLOW__KUBERNETES__DAGS_VOLUME_MOUNT_POINT: "/dags"
AIRFLOW__KUBERNETES__DAGS_VOLUME_SUBPATH: "dag/dag/"
AIRFLOW__KUBERNETES__POD_TEMPLATE_FILE: "/opt/airflow/pod_templates/pod_template.yaml"
AIRFLOW__KUBERNETES__IN_CLUSTER: "true"
AIRFLOW__KUBERNETES__NAMESPACE: "airflow-test"
AIRFLOW__KUBERNETES__WORKER_SERVICE_ACCOUNT_NAME: "airflow-test"

[thesuperzapper]

@ngreen-ny firstly, most of your AIRFLOW__KUBERNETES__XXXXX configs will not work in airflow 2.0+ (as the pod_template.yaml is the only way to do things like mounting volumes now).

Also, when using this chart you don't need to create your own pod_template.yaml, the one generated for you by the chart should automatically include everything you need. (Including automatically mounting any logs/dags volumes you have set up with the logs.persistence.* and dags.persistence.* values).

---

You can just remove all of your AIRFLOW__KUBERNETES__XXXXX configs, the chart to set up these things:

• Chart docs for Logs
• Chart docs for DAGs