[Release 1.22] Ubuntu 21.04 - vxlan failing to route

[manuelbuil]

Backport of: #4188

[mdrahman-suse]

Validated on k3s version: v1.22.3-rc3+k3s1
OS Info

$ systemd --version
systemd 247 (247.3-3ubuntu3.6)
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid

$ cat /etc/os-release
NAME="Ubuntu"
VERSION="21.04 (Hirsute Hippo)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 21.04"
VERSION_ID="21.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=hirsute
UBUNTU_CODENAME=hirsute

$ uname -a
Linux ip-172-31-36-225 5.11.0-1020-aws #21-Ubuntu SMP Wed Sep 29 11:03:00 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Steps and results

1. Deployed a k3s server node and added an k3s agent to the node
2. Applied the below yaml file

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: othertest-deploy
  labels:
    app: othertest
spec:
  selector:
    matchLabels:
      app: othertest
  template:
    metadata:
      labels:
        app: othertest
    spec:
      containers:
      - name: othertest
        image: maxross/mytestcontainer:unprivileged
        imagePullPolicy: Always

3. Exec-ed in to the pods and ping their corresponding pod ip's

$ kubectl get nodes,pods -A -o wide
NAME                    STATUS   ROLES                       AGE    VERSION            INTERNAL-IP     EXTERNAL-IP   OS-IMAGE       KERNEL-VERSION    CONTAINER-RUNTIME
node/ip-172-31-36-156   Ready    <none>                      108s   v1.22.3-rc3+k3s1   172.31.36.156   <none>        Ubuntu 21.04   5.11.0-1020-aws   containerd://1.5.7-k3s1
node/ip-172-31-36-225   Ready    control-plane,etcd,master   100m   v1.22.3-rc3+k3s1   172.31.36.225   <none>        Ubuntu 21.04   5.11.0-1020-aws   containerd://1.5.7-k3s1

NAMESPACE     NAME                                         READY   STATUS      RESTARTS   AGE    IP          NODE               NOMINATED NODE   READINESS GATES
default       pod/othertest-deploy-bll9l                   1/1     Running     0          47s    10.42.0.9   ip-172-31-36-225   <none>           <none>
default       pod/othertest-deploy-nkffd                   1/1     Running     0          47s    10.42.1.3   ip-172-31-36-156   <none>           <none>

$ kubectl exec -it othertest-deploy-bll9l -- bin/bash
nginx@othertest-deploy-bll9l:/$ ping 10.42.1.3
PING 10.42.1.3 (10.42.1.3) 56(84) bytes of data.
64 bytes from 10.42.1.3: icmp_seq=1 ttl=62 time=0.488 ms
64 bytes from 10.42.1.3: icmp_seq=2 ttl=62 time=0.455 ms
64 bytes from 10.42.1.3: icmp_seq=3 ttl=62 time=0.317 ms
^C
--- 10.42.1.3 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 40ms
rtt min/avg/max/mdev = 0.317/0.420/0.488/0.074 ms
nginx@othertest-deploy-bll9l:/$ ping 10.42.0.9
PING 10.42.0.9 (10.42.0.9) 56(84) bytes of data.
64 bytes from 10.42.0.9: icmp_seq=1 ttl=64 time=0.023 ms
64 bytes from 10.42.0.9: icmp_seq=2 ttl=64 time=0.029 ms
64 bytes from 10.42.0.9: icmp_seq=3 ttl=64 time=0.029 ms
^C
--- 10.42.0.9 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 52ms
rtt min/avg/max/mdev = 0.023/0.027/0.029/0.003 ms

$ kubectl exec -it othertest-deploy-nkffd -- bin/bash
nginx@othertest-deploy-nkffd:/$ ping 10.42.0.9
PING 10.42.0.9 (10.42.0.9) 56(84) bytes of data.
64 bytes from 10.42.0.9: icmp_seq=1 ttl=62 time=0.313 ms
64 bytes from 10.42.0.9: icmp_seq=2 ttl=62 time=0.356 ms
64 bytes from 10.42.0.9: icmp_seq=3 ttl=62 time=0.411 ms
^C
--- 10.42.0.9 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 40ms
rtt min/avg/max/mdev = 0.313/0.360/0.411/0.040 ms
nginx@othertest-deploy-nkffd:/$ ping 10.42.1.3
PING 10.42.1.3 (10.42.1.3) 56(84) bytes of data.
64 bytes from 10.42.1.3: icmp_seq=1 ttl=64 time=0.023 ms
64 bytes from 10.42.1.3: icmp_seq=2 ttl=64 time=0.029 ms
64 bytes from 10.42.1.3: icmp_seq=3 ttl=64 time=0.046 ms
^C
--- 10.42.1.3 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 53ms
rtt min/avg/max/mdev = 0.023/0.032/0.046/0.011 ms

[clemenko]

This is still broken for me.
Deploying with https://github.com/clemenko/k3s/blob/master/k3s.sh#L173

root@k3s-831b:~# systemd --version
systemd 247 (247.3-3ubuntu3)
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid

root@k3s-831b:~# cat /etc/os-release 
NAME="Ubuntu"
VERSION="21.04 (Hirsute Hippo)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 21.04"
VERSION_ID="21.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=hirsute
UBUNTU_CODENAME=hirsute

root@k3s-831b:~# uname -a
Linux k3s-831b 5.11.0-18-generic #19-Ubuntu SMP Fri May 7 14:22:03 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Then deployed the noted yaml from above.

kube:

clemair:clemenko k3s ( 167.99.124.208:6443 ) $ kubectl get nodes,pods  -o wide
NAME            STATUS   ROLES                       AGE     VERSION        INTERNAL-IP      EXTERNAL-IP   OS-IMAGE       KERNEL-VERSION      CONTAINER-RUNTIME
node/k3s-831b   Ready    control-plane,etcd,master   9m47s   v1.21.5+k3s2   167.99.124.208   <none>        Ubuntu 21.04   5.11.0-18-generic   containerd://1.4.11-k3s1
node/k3s-a95e   Ready    <none>                      9m21s   v1.21.5+k3s2   167.99.116.17    <none>        Ubuntu 21.04   5.11.0-18-generic   containerd://1.4.11-k3s1
node/k3s-bda2   Ready    <none>                      9m15s   v1.21.5+k3s2   142.93.69.32     <none>        Ubuntu 21.04   5.11.0-18-generic   containerd://1.4.11-k3s1

NAME                         READY   STATUS    RESTARTS   AGE    IP          NODE       NOMINATED NODE   READINESS GATES
pod/othertest-deploy-2glbg   1/1     Running   0          7m4s   10.42.1.7   k3s-a95e   <none>           <none>
pod/othertest-deploy-bdrbq   1/1     Running   0          7m4s   10.42.2.7   k3s-bda2   <none>           <none>
pod/othertest-deploy-blnnd   1/1     Running   0          7m4s   10.42.0.9   k3s-831b   <none>           <none>

and the pings

clemair:clemenko k3s ( 167.99.124.208:6443 ) $ kubectl exec -it othertest-deploy-2glbg -- bash
nginx@othertest-deploy-2glbg:/$ ping -c 1 -t 1 10.42.2.7   
PING 10.42.2.7 (10.42.2.7) 56(84) bytes of data.
From 10.42.1.1 icmp_seq=1 Time to live exceeded

--- 10.42.2.7 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

nginx@othertest-deploy-2glbg:/$ ping -c 1 -t 1 10.42.0.9   
PING 10.42.0.9 (10.42.0.9) 56(84) bytes of data.
From 10.42.1.1 icmp_seq=1 Time to live exceeded

--- 10.42.0.9 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

One of the things I noticed is that you are NOT using the upstream kernel. You are using the aws compiled one. Wonder if that has a fix. I am on DigitalOcean. This problem still applies to 21.10 as well.

[brandond]

As noted on the other issue where you made the same comment

QA marked this as fixed in the version that we're about to release, not the version that you're currently using. Please try again once we actually release the fixed version.