[Release-1.31] - CNI bin dir changes with K3s version

[brandond]

Backport fix for CNI bin dir changes with K3s version

• CNI bin dir changes with K3s version #10869

[aganesh-suse]

Validated on release-1.31 branch with commit 221ab22

Environment Details

Infrastructure

• Cloud
• Hosted

Node(s) CPU architecture, OS, and Version:

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04 LTS"

$ uname -m
x86_64

Cluster Configuration:

HA: 3 server/ 1 agent

Config.yaml:

token: xxxx
cluster-init: true
write-kubeconfig-mode: "0644"
node-external-ip: 1.1.1.1
node-label:
- k3s-upgrade=server

Reproduce issue multus_whereabouts_repro.yaml:

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: multus
  namespace: kube-system
spec:
  repo: https://rke2-charts.rancher.io
  chart: rke2-multus
  targetNamespace: kube-system
  valuesContent: |-
    manifests:
      configMap:
        true
    config:
      fullnameOverride: multus
      cni_conf:
        confDir: /var/lib/rancher/k3s/agent/etc/cni/net.d
        binDir: /var/lib/rancher/k3s/data/current/bin
        kubeconfig: /var/lib/rancher/k3s/agent/etc/cni/net.d/multus.d/multus.kubeconfig
    rke2-whereabouts:
      fullnameOverride: whereabouts
      enabled: true
      cniConf:
        confDir: /var/lib/rancher/k3s/agent/etc/cni/net.d
        binDir: /var/lib/rancher/k3s/data/current/bin

Validation of fix multus_whereabouts_verify.yaml:

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: multus
  namespace: kube-system
spec:
  repo: https://rke2-charts.rancher.io
  chart: rke2-multus
  targetNamespace: kube-system
  valuesContent: |-
    manifests:
      configMap:
        true
    config:
      fullnameOverride: multus
      cni_conf:
        confDir: /var/lib/rancher/k3s/agent/etc/cni/net.d
        binDir: /var/lib/rancher/k3s/data/cni
        kubeconfig: /var/lib/rancher/k3s/agent/etc/cni/net.d/multus.d/multus.kubeconfig
    rke2-whereabouts:
      fullnameOverride: whereabouts
      enabled: true
      cniConf:
        confDir: /var/lib/rancher/k3s/agent/etc/cni/net.d
        binDir: /var/lib/rancher/k3s/data/cni

Testing Steps for issue reproduction:

1. Copy config.yaml

$ sudo mkdir -p /etc/rancher/k3s && sudo cp config.yaml /etc/rancher/k3s

2. Install k3s

curl -sfL https://get.k3s.io | sudo INSTALL_K3S_VERSION='v1.31.0+k3s1' sh -s - server

3. Verify multus+whereabouts pods are coming up and the binary locations as per the applied yamls:

$ kubectl apply -f multus_whereabouts_repro.yaml

4. Upgrade to latest version.

$ curl -sfL https://get.k3s.io | sudo INSTALL_K3S_VERSION='v1.31.1+k3s1' sh -s - server

5. Check the multus + whereabouts - if they are working fine.

/var/lib/rancher/k3s/data/current/bin/multus
/var/lib/rancher/k3s/data/current/bin/whereabouts

Testing Steps for validation of fix:

1. Copy config.yaml

$ sudo mkdir -p /etc/rancher/k3s && sudo cp config.yaml /etc/rancher/k3s

2. Install k3s

For validation: 
curl -sfL https://get.k3s.io | sudo INSTALL_K3S_COMMIT='0955fa330f9008da9e1d546e66170a3abc83048f' sh -s - server

3. Verify multus+whereabouts pods are coming up and the binary locations as per the applied yamls:

$ kubectl apply -f multus_whereabouts_verify.yaml

4. Upgrade to latest commit.

curl -sfL https://get.k3s.io | sudo INSTALL_K3S_COMMIT='221ab22ca911b548d7278afb0df7fca17d2fe596' sh -s - server

5. Check the multus + whereabouts - if they are working fine.

/var/lib/rancher/k3s/data/cni/multus
/var/lib/rancher/k3s/data/cni/whereabouts

Replication Results:

• k3s version used for replication:

$ k3s -v
k3s version v1.31.1+k3s1 (452dbbc1)
go version go1.22.6

Pre-upgrade:

$ /var/lib/rancher/k3s/data/current/bin/multus 
meta-plugin that delegates to other CNI plugins
CNI protocol versions supported: 0.1.0, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 1.0.0, 1.1.0

$ /var/lib/rancher/k3s/data/current/bin/whereabouts
whereabouts v0.8.0-8c381170 linux/amd64
CNI protocol versions supported: 0.1.0, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 1.0.0, 1.1.0

Post-upgrade:

$ /var/lib/rancher/k3s/data/current/bin/multus 
bash: line 1: /var/lib/rancher/k3s/data/current/bin/multus: No such file or directory

$ /var/lib/rancher/k3s/data/current/bin/whereabouts
bash: line 1: /var/lib/rancher/k3s/data/current/bin/whereabouts: No such file or directory

Validation Results:

• k3s version used for validation:

$ k3s -v
k3s version v1.31.1+k3s-221ab22c (221ab22c)
go version go1.22.6

Pre-upgrade:

$ /var/lib/rancher/k3s/data/cni/multus 
meta-plugin that delegates to other CNI plugins
CNI protocol versions supported: 0.1.0, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 1.0.0, 1.1.0

$ /var/lib/rancher/k3s/data/cni/whereabouts
whereabouts v0.8.0-8c381170 linux/amd64
CNI protocol versions supported: 0.1.0, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 1.0.0, 1.1.0

Post-upgrade:

$ /var/lib/rancher/k3s/data/cni/multus 
meta-plugin that delegates to other CNI plugins
CNI protocol versions supported: 0.1.0, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 1.0.0, 1.1.0

$ /var/lib/rancher/k3s/data/cni/whereabouts
whereabouts v0.8.0-8c381170 linux/amd64
CNI protocol versions supported: 0.1.0, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 1.0.0, 1.1.0

Other verifications:
config.toml has the cni config and bin directory info:

$ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/config.toml | grep cni
[plugins."io.containerd.grpc.v1.cri".cni]
  bin_dir = "/var/lib/rancher/k3s/data/cni"
  conf_dir = "/var/lib/rancher/k3s/agent/etc/cni/net.d"

$ sudo ls -la /var/lib/rancher/k3s/data/ 
total 28
drwxr-xr-x 5 root root 4096 Oct 22 20:46 .
drwxr-xr-x 5 root root 4096 Oct 22 20:37 ..
-rw------- 1 root root    0 Oct 22 20:37 .lock
drwxr-xr-x 4 root root 4096 Oct 22 20:46 3d818ebb8ce4241e202a6e6a0e391c0725e6d6a0e6c1749e6fcb3dd0ed79a341
drwxr-xr-x 4 root root 4096 Oct 22 20:37 57bd7893764d682f62948413c2e32057fffc6a8a41a2e7a749d9c607c0a3cc1a
drwxr-xr-x 2 root root 4096 Oct 22 20:46 cni
lrwxrwxrwx 1 root root   90 Oct 22 20:46 current -> /var/lib/rancher/k3s/data/3d818ebb8ce4241e202a6e6a0e391c0725e6d6a0e6c1749e6fcb3dd0ed79a341
lrwxrwxrwx 1 root root   90 Oct 22 20:37 previous -> /var/lib/rancher/k3s/data/57bd7893764d682f62948413c2e32057fffc6a8a41a2e7a749d9c607c0a3cc1a

$ sudo ls -la /var/lib/rancher/k3s/data/cni 
total 159908
drwxr-xr-x 2 root root     4096 Oct 22 20:46 .
drwxr-xr-x 5 root root     4096 Oct 22 20:46 ..
-rwxr-xr-x 1 root root  5044864 Oct 22 20:45 bandwidth
-rwxr-xr-x 1 root root  5480992 Oct 22 20:45 bridge
lrwxrwxrwx 1 root root       98 Oct 22 20:46 cni -> /var/lib/rancher/k3s/data/3d818ebb8ce4241e202a6e6a0e391c0725e6d6a0e6c1749e6fcb3dd0ed79a341/bin/cni
-rwxr-xr-x 1 root root 10813312 Oct 22 20:45 dhcp
-rwxr-xr-x 1 root root  5177248 Oct 22 20:45 dummy
-rwxr-xr-x 1 root root  5509312 Oct 22 20:45 firewall
lrwxrwxrwx 1 root root       98 Oct 22 20:46 flannel -> /var/lib/rancher/k3s/data/3d818ebb8ce4241e202a6e6a0e391c0725e6d6a0e6c1749e6fcb3dd0ed79a341/bin/cni
-rwxr-xr-x 1 root root  5120000 Oct 22 20:45 host-device
-rwxr-xr-x 1 root root  4614752 Oct 22 20:45 host-local
-rwxr-xr-x 1 root root  5185440 Oct 22 20:45 ipvlan
-rwxr-xr-x 1 root root  2736912 Oct 22 20:45 loopback
-rwxr-xr-x 1 root root  5210048 Oct 22 20:45 macvlan
-rwxr-xr-x 1 root root 39183128 Oct 22 20:45 multus
-rwxr-xr-x 1 root root  5078848 Oct 22 20:45 portmap
-rwxr-xr-x 1 root root  5329472 Oct 22 20:45 ptp
-rwxr-xr-x 1 root root  2893264 Oct 22 20:45 sbr
-rwxr-xr-x 1 root root  2428240 Oct 22 20:45 static
-rwxr-xr-x 1 root root  5232128 Oct 22 20:45 tap
-rwxr-xr-x 1 root root  2798512 Oct 22 20:45 tuning
-rwxr-xr-x 1 root root  5185440 Oct 22 20:45 vlan
-rwxr-xr-x 1 root root  3001104 Oct 22 20:45 vrf
-rwxr-xr-x 1 root root 37671392 Oct 22 20:45 whereabouts

$ sudo ls -lrt /var/lib/rancher/k3s/agent/etc/cni/net.d 
total 16
drwxr-xr-x 2 root root 4096 Oct 22 20:45 whereabouts.d
drwxr-xr-x 2 root root 4096 Oct 22 20:45 multus.d
-rw------- 1 root root  623 Oct 22 20:45 00-multus.conflist
-rw-r--r-- 1 root root  406 Oct 22 20:46 10-flannel.conflist

Verifications for the CNI symlink exists fatal error:

To reproduce: Upgrade from commit 5159131 to commit 0955fa3
To validate: Upgrade from commit 5159131 to commit 221ab22
Test steps: #10869 (comment)
On a repro setup, you will see logs:

$ journalctl -xeu k3s | grep 'fatal' 
Oct 23 16:22:56 ip-172-31-3-119 k3s[104425]: time="2024-10-23T16:22:56Z" level=fatal msg="extracting data: symlink /var/lib/rancher/k3s/data/57bd7893764d682f62948413c2e32057fffc6a8a41a2e7a749d9c607c0a3cc1a/bin/cni /var/lib/rancher/k3s/data/cni/cni: file exists"

On the validated setup:
No fatal errors on all servers/agents:

$ journalctl -xeu k3s | grep 'fatal'