Need advice/help - kubernetes deployment, gitSync fails to connect to gitlab self-hosted repo

[hanleybrand]

I've been trying to switch my airflow deployment from the dags baked into the deployment image to gitSync, mounting the dags without persistence (without success

I have a kubernetes secret secrets-airflow created in the cluster in the airflow namespace, and it contains base64 encoded keys as described in the helm chart values.yaml (GITLAB_ACCESS_TOKEN is the un-base64 version of GIT_SYNC_PASSWORD)

k describe secrets/secrets-airflow -n airflow | grep -i git
GITLAB_ACCESS_TOKEN:                      26 bytes
GIT_SYNC_USERNAME:                        16 bytes
GIT_SYNC_PASSWORD:                        36 bytes
gitSshKey:                                548 bytes

If I execute a shell on the scheduler pod, and do either of the following tests:

pod shell: scheduler test 1 (GIT_SYNC_USERNAME/PASSWORD)

USERNAME=$(echo $GIT_SYNC_USERNAME | openssl enc -d -base64)
git clone -b dev https://$USERNAME:$GITLAB_ACCESS_TOKEN@git.temple.edu/academic-apps/airflow.git  /tmp/af

pod shell: scheduler test 2 (gitSshKey)

echo $gitSshKey | openssl enc -d -base64 > gitSshKey.txt
chmod 600 ./gitSshKey.txt
# both of the below work
GIT_SSH_COMMAND='ssh -i ./gitSshKey' git clone [email protected]:academic-apps/airflow.git /tmp/af2
GIT_SSH_COMMAND='ssh -i ./gitSshKey.txt' git clone ssh://[email protected]/academic-apps/airflow.git af3

Both tests (all three) cloned the repo, which suggests to me that the correct information is being passed to the gitSync sidecar -- when I did the second test I had to manually accept the connection/add my gitlab server to known hosts*

So here's my best determination of what the config should be

dags:
  persistence:
    enabled: false
  gitSync:
    enabled: true
    repo: [email protected]:academic-apps/airflow.git
    branch: dev
    depth: 1
    maxFailures: 0
    subPath: "dags"
    sshKeySecret: secrets-airflow
    # has base64 encoded private key in gitSshKey 
    knownHosts: |
        git.temple.edu ecdsa-sha2-nistp256 AAAA...imk=

this results in the following "Load key "/etc/git-secret/ssh": invalid format" error in the git-sync-init container on deploy with either the above dags.gitSync.repo value, or with a value of ssh://[email protected]/academic-apps/airflow.git as some example I saw had:

"level"=0 "msg"="starting up" "pid"=13 "args"=["/git-sync"] 
"level"=0 "msg"="cloning repo" "origin"="[email protected]:academic-apps/airflow.git" "path"="/git" 
"msg"="too many failures, aborting" "error"="Run(git clone --no-checkout -b dev --depth 1 [email protected]:academic-apps/airflow.git /git): exit status 128: { stdout: "", stderr: "Cloning into '/git'...
Load key \"/etc/git-secret/ssh\": invalid format
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights and the repository exists." }" "failCount"=0 

So I'm guessing the likely solution will be to figure out what's making the key format invalid, but I'm not sure where to go from here -- in case it's relevant, I initially encoded the key like so:

base64 -i -b ~/.ssh/git_temple_edu -o ~/.ssh/git_temple_edu_b64

I'm assuming I missed something dumb, but I can't figure out what it might be.

The other way

I also tried it with basic auth/username/password via https) with the following changes:

dags:
  gitSync:
    repo: https://git.temple.edu/academic-apps/airflow.git
    credentialsSecret: secrets-airflow
    # has base64 encoded GIT_SYNC_USERNAME/GIT_SYNC_PASSWORD

I get a different error (could not read Username for 'https://git.temple.edu': No such device or address):

"level"=0 "msg"="starting up" "pid"=11 "args"=["/git-sync"] 
"level"=0 "msg"="cloning repo" "origin"="[email protected]:academic-apps/airflow.git" "path"="/git" 
 "msg"="too many failures, aborting" 
"error"="Run(git clone --no-checkout -b dev --depth 1 https://git.temple.edu/academic-apps/airflow.git /git): 
exit status 128: { stdout: "", stderr: "Cloning into '/git'...
fatal: could not read Username for 'https://git.temple.edu': No such device or address" }" "failCount"=0 

Is it something dumb like I shouldn't be base64 encoding the secrets?

[kobethuwis]

I'm using Airflow 2.5.1 with chart version 1.8.0 with git-sync as well, I'll share my config:

dags:
  gitSync:
    enabled: true
    repo: ${var.dags_repo_url} #https://myrepo/myself/repo.git
    branch: ${var.dags_branch} #main
    subPath: dags #dags are located in /dags
    credentialsSecret: my-gitsync-credentials

resource "kubernetes_secret" "my-gitsync-credentials" {

  metadata {
    name      = "my-gitsync-credentials"
    namespace = "mynamespace"
  }
  data = {
    GIT_SYNC_USERNAME = "plain_username"
    GIT_SYNC_PASSWORD = "plain_password"
  }
}

If I remember correctly the data inside the k8s secret is indeed base64 encoded so you shouldn't have to do that yourself. Haven't tried with an ssh key though

[hanleybrand]

Thanks for your response -- I changed the secret to its un-encoded value, and the error is slightly different. I'm starting to think this is a permissions issue vs the secret not being correct

"msg"="too many failures, aborting" 
"error"="Run(git clone --no-checkout -b dev --depth 1 [email protected]:academic-apps/airflow.git /git): 
exit status 128: { stdout: "", stderr: "Cloning into '/git'...
Failed to add the ECDSA host key for IP address '155.247.xxx.xxx' to the list of known hosts (/etc/git-secret/known_hosts).
Load key "/etc/git-secret/ssh": invalid format
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights and the repository exists." }" "failCount"=0 

I had not previously thought about the values after dags.gitSync.knownhosts:

    # knownHosts: |
    #    <host1>,<ip1> <key1>
    #    <host2>,<ip2> <key2>

    wait: 5
    containerName: git-sync
    uid: 65533
    # When not set, the values defined in the global securityContext will be used  
    securityContext: {}  
    extraVolumeMounts: []  
    env: []  
    resources: {}

I've been reviewing the git-sync repo to try to glean some info, alas they don't have Discussions open, but according to the git-sync Dockerfile:

# This container will run as UID:GID 65533:65533 by default. For most users,
# the simplest ways to use this container are either:
# a) use the default UID/GID and mount a volume on /git writeable by those
# b) set your own UID/GID and mount a volume on /git writeable by those
#
# If you mount a volume anywhere else, you must set `--root` ($GIT_SYNC_ROOT).
# If you do not mount a volume, this will run but you can't access the results
# (which might be useful for testing, but not much else).
#
# Newly created docker volumes (the very first `docker run -v`) are initialized
# based on the in-image mountpoint's UID/GID?mode, so another solution for
# using this with docker is to set your own UID/GID and also add the default
# GID as a supplemental group (see `docker run --group-add`).
#
# Kubernetes offers `Pod.spec.securityContext.fsGroup` to manage volume
# permissions.
#
# If you set any UID other than the default and want to use git over SSH, you
# should set `--add-user` ($GIT_SYNC_ADD_USER).

I don't really know how that helps me, though. I'm going to start by going to a meeting to clear my head and then I'll try forgoing knownhosts to see if I can get the git clone to work at all.

Thanks again!

[hanleybrand]

Update: After going down the rabbit hole of permissions, how gitsync works with hammerspace-nfs volumes and PVCs (still not entirely sure about that one), initContainers and allthethings, I ended up reading through the official Dockerfile and discovering that I had the airflow user running as as uid/gid 1000:1000 instead of 50000:0 which in retrospect is mentioned pretty clearly in the Dockerfile 🤭

So it's working now, pretty well actually

Dockerfile

USER root  

SHELL ["/bin/bash", "-o", "pipefail", "-c"]
RUN adduser --gecos "First Last,RoomNumber,WorkPhone,HomePhone" --disabled-password \
           --quiet "airflow" --uid  $AIRFLOW_UID --gid 0 --home "${AIRFLOW_USER_HOME_DIR}" \
    && mkdir -pv "${AIRFLOW_HOME}" \
    && chown -R airflow:0 "${AIRFLOW_USER_HOME_DIR}" "${AIRFLOW_HOME}" \
    && chmod g=u /etc/passwd \
    && echo "It's like 50000 or 0 better" \
    && PY_MAJ_MIN="$(python --version | cut -d ' ' -f 2 | cut -d '.' -f 1-2)" \
    && export PY_MAJ_MIN \
    && PYTHONPATH="${AIRFLOW_USER_HOME_DIR}/.local/lib/python${PY_MAJ_MIN}/site-packages:${PYTHONPATH:-}" \
    && export PYTHONPATH

USER airflow

ENV PY_MIN_MAJ=$PY_MIN_MAJ
ENV PATH="${AIRFLOW_USER_HOME_DIR}/.local/bin:$PATH"
ENV PYTHONPATH=$PYTHONPATH

values.yml

dags:
  persistence:
    enabled: false
  gitSync:
    enabled: true
    repo: https://git.temple.edu/academic-apps/airflow.git  # gitlab self-hosted
    branch: dev
    rev: HEAD
    depth: 1
    maxFailures: 10
    subPath: "dags"
    credentialsSecret: secrets-airflow
    wait: 60
    containerName: git-sync
    uid: 50000