the overlay network between nodes seems to be broken in Debian 11 (bullseye) #3863

rgl · 2021-08-15T19:57:26Z

Environmental Info:

K3s Version:

k3s version v1.21.3+k3s1 (1d1f220f)
go version go1.16.6

Node(s) CPU architecture, OS, and Version:

root@s1:~# uname -a
Linux s1 5.10.0-8-amd64 #1 SMP Debian 5.10.46-4 (2021-08-03) x86_64 GNU/Linux
root@s1:~# kubectl get nodes -o wide
NAME   STATUS   ROLES                  AGE   VERSION        INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION   CONTAINER-RUNTIME
s1     Ready    control-plane,master   17m   v1.21.3+k3s1   10.11.0.101   <none>        Debian GNU/Linux 11 (bullseye)   5.10.0-8-amd64   containerd://1.4.8-k3s1
a1     Ready    <none>                 14m   v1.21.3+k3s1   10.11.0.201   <none>        Debian GNU/Linux 11 (bullseye)   5.10.0-8-amd64   containerd://1.4.8-k3s1

Cluster Configuration:

1 server (without worker role) and 1 agent as configured by my playground at https://github.com/rgl/k3s-vagrant/tree/debian-11-wip.

Describe the bug:

the traefik dashboard is installed as https://github.com/rgl/k3s-vagrant/blob/debian-11-wip/provision-k3s-server.sh#L62-L167 and is available in all the cluster nodes at:

http://10.11.0.101:9000/dashboard/ (s1 node)
http://10.11.0.201:9000/dashboard/ (a1 node)

when running in Debian 10, accessing both addresses works fine (they return the expected web page). but when running in Debian 11, the a1 node address does not work (the request times out).

it does not even work when trying a wget http://10.11.0.201:9000/dashboard/ from the s1 node (but works inside the a1 node).

while running tcpdump at the s1 and a1 nodes, I can see the SYN packets leave s1 to a1, but a1 never replies.

do you have any idea why this is happening or what might be blocking this? or any clue how to make it work?

PS when launching the server with --flannel-backend 'host-gw' things seem to work. so it seems there's something going on with the vxlan backend.

Steps To Reproduce:

install k3s like https://github.com/rgl/k3s-vagrant/blob/wip-vxlan/provision-k3s-server.sh#L46-L61

PS: That is running in KVM VMs (using virtio nic) on a Ubuntu 20.04 host. The code is at https://github.com/rgl/k3s-vagrant/tree/wip-vxlan.

Expected behavior:

expected it to work in Debian 11, like it does in Debian 10.

Actual behavior:

traffic between nodes in the overlay network does not seem to be working correctly.

Additional context / logs:

Debian 11 ships by default with cgroup2, but I've switched to cgroup1 with https://github.com/rgl/k3s-vagrant/blob/debian-11-wip/provision-cgroup1.sh

The text was updated successfully, but these errors were encountered:

brandond · 2021-08-17T18:09:47Z

I've seen other issues with iptables on Debian, for example #3117 (comment) - I wonder if this is related?

rgl · 2021-08-22T10:15:32Z

upgraded to v1.21.4+k3s1 and its working fine now.

Nothing4You · 2021-08-22T12:13:46Z

I'm still seeing this problem on v1.21.4+k3s1 on bulleseye, in my case 2 Hetzner cloud instances.
For testing I just increased coredns to 2 replicas to have one on each node, then I tested DNS resolution via gcr.io/kubernetes-e2e-test-images/dnsutils:1.3, querying both the local node endpoint and the remote node endpoint.

kubectl --context test-hcloud-k3s get pods -A -o wide

NAMESPACE     NAME                                      READY   STATUS    RESTARTS   AGE     IP          NODE                   NOMINATED NODE   READINESS GATES
kube-system   local-path-provisioner-5ff76fc89d-pv8nx   1/1     Running   0          10m     10.42.0.2   test-hc-k3s-master-0   <none>           <none>
kube-system   coredns-7448499f4d-tcb8g                  1/1     Running   0          10m     10.42.0.4   test-hc-k3s-master-0   <none>           <none>
kube-system   metrics-server-86cbb8457f-8rq25           1/1     Running   0          10m     10.42.0.3   test-hc-k3s-master-0   <none>           <none>
kube-system   coredns-7448499f4d-vdkv5                  1/1     Running   0          6m21s   10.42.1.2   test-hc-k3s-agent-0    <none>           <none>
default       dnsutils                                  1/1     Running   0          5m55s   10.42.0.5   test-hc-k3s-master-0   <none>           <none>

kubectl --context test-hcloud-k3s describe endpoints kube-dns -n kube-system

Name:         kube-dns
Namespace:    kube-system
Labels:       k8s-app=kube-dns
              kubernetes.io/cluster-service=true
              kubernetes.io/name=CoreDNS
              objectset.rio.cattle.io/hash=bce283298811743a0386ab510f2f67ef74240c57
Annotations:  endpoints.kubernetes.io/last-change-trigger-time: 2021-08-22T13:44:03+02:00
Subsets:
  Addresses:          10.42.0.4,10.42.1.2
  NotReadyAddresses:  <none>
  Ports:
    Name     Port  Protocol
    ----     ----  --------
    dns-tcp  53    TCP
    dns      53    UDP
    metrics  9153  TCP

Events:  <none>

Now when I try to query the pod on the master node from the dnsutils container on the master node everything is working fine:
kubectl --context test-hcloud-k3s exec -it dnsutils -- dig +search kube-dns.kube-system @10.42.0.4

; <<>> DiG 9.11.6-P1 <<>> +search kube-dns.kube-system @10.42.0.4
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49634
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: fd2f9e9d0725fd6b (echoed)
;; QUESTION SECTION:
;kube-dns.kube-system.svc.cluster.local.	IN A

;; ANSWER SECTION:
kube-dns.kube-system.svc.cluster.local.	5 IN A	10.43.0.10

;; Query time: 0 msec
;; SERVER: 10.42.0.4#53(10.42.0.4)
;; WHEN: Sun Aug 22 11:51:21 UTC 2021
;; MSG SIZE  rcvd: 133

root@test-hc-k3s-master-0:~# tcpdump -i any -nn port 53

13:51:21.306554 vethe10710cb P   IP 10.42.0.5.58375 > 10.42.0.4.53: 8851+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
13:51:21.306624 veth5e515acc Out IP 10.42.0.5.58375 > 10.42.0.4.53: 8851+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
13:51:21.307048 veth5e515acc P   IP 10.42.0.4.53 > 10.42.0.5.58375: 8851 NXDomain*- 0/1/1 (180)
13:51:21.307068 vethe10710cb Out IP 10.42.0.4.53 > 10.42.0.5.58375: 8851 NXDomain*- 0/1/1 (180)
13:51:21.307295 vethe10710cb P   IP 10.42.0.5.40611 > 10.42.0.4.53: 49634+ [1au] A? kube-dns.kube-system.svc.cluster.local. (79)
13:51:21.307322 veth5e515acc Out IP 10.42.0.5.40611 > 10.42.0.4.53: 49634+ [1au] A? kube-dns.kube-system.svc.cluster.local. (79)
13:51:21.307477 veth5e515acc P   IP 10.42.0.4.53 > 10.42.0.5.40611: 49634*- 1/0/1 A 10.43.0.10 (133)
13:51:21.307484 vethe10710cb Out IP 10.42.0.4.53 > 10.42.0.5.40611: 49634*- 1/0/1 A 10.43.0.10 (133)

Now let's repeat the same, querying the pod on the agent node from the dnstest container on the master node:
kubectl --context test-hcloud-k3s exec -it dnsutils -- dig +search kube-dns.kube-system @10.42.1.2

; <<>> DiG 9.11.6-P1 <<>> +search kube-dns.kube-system @10.42.1.2
;; global options: +cmd
;; connection timed out; no servers could be reached
command terminated with exit code 9

root@test-hc-k3s-master-0:~# tcpdump -i any -nn port 53

13:57:15.906884 vethe10710cb P   IP 10.42.0.5.38892 > 10.42.1.2.53: 58479+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
13:57:15.906884 cni0  In  IP 10.42.0.5.38892 > 10.42.1.2.53: 58479+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
13:57:15.906942 flannel.1 Out IP 10.42.0.5.38892 > 10.42.1.2.53: 58479+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
13:57:15.909608 flannel.1 P   IP 10.42.1.2.53 > 10.42.0.5.38892: 58479 NXDomain*- 0/1/1 (180)
13:57:20.906768 vethe10710cb P   IP 10.42.0.5.38892 > 10.42.1.2.53: 58479+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
13:57:20.906768 cni0  In  IP 10.42.0.5.38892 > 10.42.1.2.53: 58479+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
13:57:20.906877 flannel.1 Out IP 10.42.0.5.38892 > 10.42.1.2.53: 58479+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
13:57:20.907807 flannel.1 P   IP 10.42.1.2.53 > 10.42.0.5.38892: 58479 NXDomain*- 0/1/1 (180)
13:57:25.906785 vethe10710cb P   IP 10.42.0.5.38892 > 10.42.1.2.53: 58479+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
13:57:25.906785 cni0  In  IP 10.42.0.5.38892 > 10.42.1.2.53: 58479+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
13:57:25.906828 flannel.1 Out IP 10.42.0.5.38892 > 10.42.1.2.53: 58479+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
13:57:25.908381 flannel.1 P   IP 10.42.1.2.53 > 10.42.0.5.38892: 58479 NXDomain*- 0/1/1 (180)

root@test-hc-k3s-agent-0:~# tcpdump -i any -nn port 53

13:57:15.907722 flannel.1 In  IP 10.42.0.5.38892 > 10.42.1.2.53: 58479+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
13:57:15.907779 cni0  Out IP 10.42.0.5.38892 > 10.42.1.2.53: 58479+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
13:57:15.907789 vethb0b8faff Out IP 10.42.0.5.38892 > 10.42.1.2.53: 58479+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
13:57:15.909179 vethb0b8faff P   IP 10.42.1.2.53 > 10.42.0.5.38892: 58479 NXDomain*- 0/1/1 (180)
13:57:15.909196 cni0  In  IP 10.42.1.2.53 > 10.42.0.5.38892: 58479 NXDomain*- 0/1/1 (180)
13:57:15.909239 flannel.1 Out IP 10.42.1.2.53 > 10.42.0.5.38892: 58479 NXDomain*- 0/1/1 (180)
13:57:20.907678 flannel.1 In  IP 10.42.0.5.38892 > 10.42.1.2.53: 58479+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
13:57:20.907698 cni0  Out IP 10.42.0.5.38892 > 10.42.1.2.53: 58479+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
13:57:20.907707 vethb0b8faff Out IP 10.42.0.5.38892 > 10.42.1.2.53: 58479+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
13:57:20.908064 vethb0b8faff P   IP 10.42.1.2.53 > 10.42.0.5.38892: 58479 NXDomain*- 0/1/1 (180)
13:57:20.908097 cni0  In  IP 10.42.1.2.53 > 10.42.0.5.38892: 58479 NXDomain*- 0/1/1 (180)
13:57:20.908113 flannel.1 Out IP 10.42.1.2.53 > 10.42.0.5.38892: 58479 NXDomain*- 0/1/1 (180)
13:57:25.908036 flannel.1 In  IP 10.42.0.5.38892 > 10.42.1.2.53: 58479+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
13:57:25.908062 cni0  Out IP 10.42.0.5.38892 > 10.42.1.2.53: 58479+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
13:57:25.908074 vethb0b8faff Out IP 10.42.0.5.38892 > 10.42.1.2.53: 58479+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
13:57:25.908900 vethb0b8faff P   IP 10.42.1.2.53 > 10.42.0.5.38892: 58479 NXDomain*- 0/1/1 (180)
13:57:25.908918 cni0  In  IP 10.42.1.2.53 > 10.42.0.5.38892: 58479 NXDomain*- 0/1/1 (180)
13:57:25.908962 flannel.1 Out IP 10.42.1.2.53 > 10.42.0.5.38892: 58479 NXDomain*- 0/1/1 (180)

It does seem to be an improvement over v1.21.3+k3s1 though as with v1.21.3+k3s1 I don't even see a response in tcpdump at all:

kubectl --context test-hcloud-k3s get pods -A -o wide

NAMESPACE     NAME                                      READY   STATUS    RESTARTS   AGE     IP          NODE                   NOMINATED NODE   READINESS GATES
kube-system   coredns-7448499f4d-zb92q                  1/1     Running   0          8m31s   10.42.0.2   test-hc-k3s-master-0   <none>           <none>
kube-system   local-path-provisioner-5ff76fc89d-5gw2r   1/1     Running   0          8m31s   10.42.0.3   test-hc-k3s-master-0   <none>           <none>
kube-system   metrics-server-86cbb8457f-gsghc           1/1     Running   0          8m31s   10.42.0.4   test-hc-k3s-master-0   <none>           <none>
default       dnsutils                                  1/1     Running   0          6m26s   10.42.0.5   test-hc-k3s-master-0   <none>           <none>
kube-system   coredns-7448499f4d-qpwk8                  1/1     Running   0          5m43s   10.42.1.2   test-hc-k3s-agent-0    <none>           <none>

kubectl --context test-hcloud-k3s describe endpoints kube-dns -n kube-system

Name:         kube-dns
Namespace:    kube-system
Labels:       k8s-app=kube-dns
              kubernetes.io/cluster-service=true
              kubernetes.io/name=CoreDNS
              objectset.rio.cattle.io/hash=bce283298811743a0386ab510f2f67ef74240c57
Annotations:  endpoints.kubernetes.io/last-change-trigger-time: 2021-08-22T14:04:53+02:00
Subsets:
  Addresses:          10.42.0.2,10.42.1.2
  NotReadyAddresses:  <none>
  Ports:
    Name     Port  Protocol
    ----     ----  --------
    dns-tcp  53    TCP
    dns      53    UDP
    metrics  9153  TCP

Events:  <none>

local request within master node:
root@test-hc-k3s-master-0:~# tcpdump -i any -nn port 53

14:05:32.404481 veth7c207c5b P   IP 10.42.0.5.53914 > 10.42.0.2.53: 63669+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
14:05:32.404578 vethdb968d68 Out IP 10.42.0.5.53914 > 10.42.0.2.53: 63669+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
14:05:32.405599 vethdb968d68 P   IP 10.42.0.2.53 > 10.42.0.5.53914: 63669 NXDomain*- 0/1/1 (180)
14:05:32.405626 veth7c207c5b Out IP 10.42.0.2.53 > 10.42.0.5.53914: 63669 NXDomain*- 0/1/1 (180)
14:05:32.407124 veth7c207c5b P   IP 10.42.0.5.46851 > 10.42.0.2.53: 18114+ [1au] A? kube-dns.kube-system.svc.cluster.local. (79)
14:05:32.407200 vethdb968d68 Out IP 10.42.0.5.46851 > 10.42.0.2.53: 18114+ [1au] A? kube-dns.kube-system.svc.cluster.local. (79)
14:05:32.407557 vethdb968d68 P   IP 10.42.0.2.53 > 10.42.0.5.46851: 18114*- 1/0/1 A 10.43.0.10 (133)
14:05:32.407568 veth7c207c5b Out IP 10.42.0.2.53 > 10.42.0.5.46851: 18114*- 1/0/1 A 10.43.0.10 (133)

request from master node to agent node:
root@test-hc-k3s-master-0:~# tcpdump -i any -nn port 53

14:05:38.526888 veth7c207c5b P   IP 10.42.0.5.50776 > 10.42.1.2.53: 59600+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
14:05:38.526888 cni0  In  IP 10.42.0.5.50776 > 10.42.1.2.53: 59600+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
14:05:38.526958 flannel.1 Out IP 10.42.0.5.50776 > 10.42.1.2.53: 59600+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
14:05:43.526755 veth7c207c5b P   IP 10.42.0.5.50776 > 10.42.1.2.53: 59600+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
14:05:43.526755 cni0  In  IP 10.42.0.5.50776 > 10.42.1.2.53: 59600+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
14:05:43.526803 flannel.1 Out IP 10.42.0.5.50776 > 10.42.1.2.53: 59600+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
14:05:48.527231 veth7c207c5b P   IP 10.42.0.5.50776 > 10.42.1.2.53: 59600+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
14:05:48.527231 cni0  In  IP 10.42.0.5.50776 > 10.42.1.2.53: 59600+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
14:05:48.527369 flannel.1 Out IP 10.42.0.5.50776 > 10.42.1.2.53: 59600+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)

root@test-hc-k3s-agent-0:~# tcpdump -i any -nn port 53

14:05:38.519772 flannel.1 P   IP 10.42.0.5.50776 > 10.42.1.2.53: 59600+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
14:05:43.519393 flannel.1 P   IP 10.42.0.5.50776 > 10.42.1.2.53: 59600+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)
14:05:48.520172 flannel.1 P   IP 10.42.0.5.50776 > 10.42.1.2.53: 59600+ [1au] A? kube-dns.kube-system.default.svc.cluster.local. (87)

rgl · 2021-08-22T13:23:00Z

Indeed, it has semmed to work. But I've recreated the environment again, and it does not work. Nor does it work with the v1.22.1-rc1+k3s1.

rgl · 2021-08-22T20:29:19Z

when launching the server with --flannel-backend 'host-gw' things seem to work. so it seems there's something going on with the vxlan backend.

brandond · 2021-08-23T04:06:44Z

There have been a bunch of issues with udp checksum offload in recent kernels, I suspect that's the root cause here.

rgl · 2021-08-23T19:10:04Z

@brandond do you have any links about that? any idea if there's an workaround?

brandond · 2021-08-24T07:23:20Z

Not sure why people keep tagging @bradtopol in stuff...

rgl · 2021-08-24T08:45:41Z

@brandond, sorry! I didn't pay attention. I really wanted to tag you.

What happened in my case, is that for some odd reason your name is not the first on the github code completion. It seems it doesn't rank the people that are already in a issue to appear first.

Nothing4You · 2021-08-24T19:40:31Z

disabling offloading via ethtool -K eth0 rx off tx off did not seem to help in my case.

it also doesn't seem to be fully consistent, for this testing i mostly kept rebuilding vms at hetzner cloud with varying results.
it seems that i can get both behaviors listed above happen which i previously attributed to different k3s versions and in some circumstances which i wasn't able to identify in a reproducible way yet it was also working just fine.

it does however as @rgl already mentioned seem to be an issue that only affects vxlan.
as wireguard is part of bullseye/stable now i've given that a try and it seems to work just fine for me.

brandond · 2021-08-26T03:17:30Z

You need to disable checksum on the vxlan interface, not the physical interface. See:

rgl · 2021-08-26T06:32:35Z

According to projectcalico/felix#2811 the offload bug only affects Linux <5.7. Debian 11 is using Linux 5.10.

Even so, I gave it a try, but it did not work here. Something else seems to be braking vxlan :-(

Here's how I've disable it on all machine and even all interfaces:

for i in eth0 eth1 flannel.1 cni0; do
  #ethtool -K $i tx-checksum-ip-generic off
  ethtool -K $i tx off rx off
done

rgl · 2021-08-26T06:40:49Z

I forgot to make this more explicit before, but I'm running this in KVM VMs (using virtio nic) on a Ubuntu 20.04 host. The code is at https://github.com/rgl/k3s-vagrant/tree/wip-vxlan.

jesseward · 2021-08-27T20:04:42Z

same encountered in my recently migrated Debian 11 (kernel 5.10.46 env), details are..

Linux k3s-node01 5.10.0-8-amd64 #1 SMP Debian 5.10.46-4 (2021-08-03) x86_64 GNU/Linux
...
NAME         STATUS   ROLES                  AGE   VERSION
k3s-node01   Ready    control-plane,master   20m   v1.21.4+k3s1
k3s-node02   Ready    <none>                 19m   v1.21.4+k3s1

removing vxlan dependency and launching k3s server with --flannel-backend 'host-gw' confirmed as a workaround..

Edit: I have grabbed a pair of packet captures on the Debian 11 hosts. Confirmed what others have mentioned, only traffic from the Master between the Overlay network fails. But the nodes/agents can communicate without issue over the flannel network.

hostname	flannel.1 address	eno1
k3s-master01	10.42.0.0/32	192.168.1.16
k3s-node01	10.42.2.0/32	192.168.1.17
k3s-node02	10.42.1.0/32	192.168.1.18

On k3s-master01 and k3s-node02 , I generated traffic to the certmanager pod, that resides on k3s-node01 (10.42.2.0) , by doing nc -v 10.43.133.142 443 . Traffic was captured at the destination (k3s-node01).

packet captures zipped up below..

vxlan.good.zip - k3s-node02 -> k3s-node01 - Handshake completes, all is well from node->node communication.
vxlan.bad.zip - k3s-master-02 -> k3s-node01 . What's interesting is that I see the UDP payload arrive at the host, and it appears to be encapsulated appropriately, the underlying TCP packet is a SYN. But there is no response. SYNs are resent.

I wonder if this is an iptables issue with the source address range used by the master (10.42.0.1/24) ? Though on the destinations (k3s-node[12]) i see the following..

ACCEPT     all  --  10.42.0.0/16         0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            10.42.0.0/16

@brandond : Is it possible for me to manually assign the flannel network range on my master something other than 10.42.0.1/2 ?

brandond · 2021-08-27T20:14:51Z

Sure sounds like something's up with the Debian 11 vxlan kernel module...

Nothing4You · 2021-08-31T16:10:44Z

@jesseward as per https://rancher.com/docs/k3s/latest/en/installation/install-options/server-config/#networking that should easily be doable

ikaruswill · 2021-10-01T11:33:09Z

+1 I've encountered the same issue on a vanilla installation of Bullseye on an NUC7PJYH.

nilbacardit26 · 2021-10-04T02:23:49Z

Same here. I've got a cluster made of rapsberrys and one ASUS NP-51 running with debian 11. The one overlay network seems to fail only for the node running with debian. The connectivity between the rapsberrys is just fine.

brandond · 2021-10-18T19:27:26Z

Duplicate of #4188 (comment) - we'll track the fix over there.

homerzhou · 2023-02-08T00:46:17Z

@rgl I know the reason why vxlan can not connect in debian11.

Every vxlan interface create in debian11 will be have the same mac address.

rgl · 2023-02-09T22:36:07Z

@homerzhou, in the meantime this issue was fixed and vxlan is working in debian 11.

This was referenced Aug 15, 2021

Handle cgroup v1/2/hybrid in check-config.sh more explicitly/accurately #3230

Merged

K3s does not start on Debian Bullseye #3224

Closed

rgl closed this as completed Aug 22, 2021

rgl reopened this Aug 22, 2021

Nothing4You mentioned this issue Aug 31, 2021

CrashLoopBackOff on deploying the service keel-hq/keel#532

Open

Nothing4You mentioned this issue Sep 9, 2021

Support Debian Bullseye PyratLabs/ansible-role-k3s#151

Closed

brandond closed this as completed Oct 18, 2021

lossos mentioned this issue Oct 19, 2021

Flannel fails to communicate between pods after node reboot flannel-io/flannel#1474

Closed

Brice187 mentioned this issue Oct 29, 2021

Ubuntu 21.04 - vxlan failing to route #4188

Closed

svenja11 mentioned this issue Aug 8, 2023

How to set up K3S, GlusterFS and Hetzner's cloud load balancer hetzneronline/community-content#662

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

the overlay network between nodes seems to be broken in Debian 11 (bullseye) #3863

the overlay network between nodes seems to be broken in Debian 11 (bullseye) #3863

rgl commented Aug 15, 2021 •

edited

Loading

brandond commented Aug 17, 2021

rgl commented Aug 22, 2021

Nothing4You commented Aug 22, 2021

rgl commented Aug 22, 2021

rgl commented Aug 22, 2021 •

edited

Loading

brandond commented Aug 23, 2021

rgl commented Aug 23, 2021 •

edited

Loading

brandond commented Aug 24, 2021

rgl commented Aug 24, 2021

Nothing4You commented Aug 24, 2021

brandond commented Aug 26, 2021 •

edited

Loading

rgl commented Aug 26, 2021

rgl commented Aug 26, 2021 •

edited

Loading

jesseward commented Aug 27, 2021 •

edited

Loading

brandond commented Aug 27, 2021

Nothing4You commented Aug 31, 2021

ikaruswill commented Oct 1, 2021

nilbacardit26 commented Oct 4, 2021

brandond commented Oct 18, 2021 •

edited

Loading

homerzhou commented Feb 8, 2023

rgl commented Feb 9, 2023 •

edited

Loading

the overlay network between nodes seems to be broken in Debian 11 (bullseye) #3863

the overlay network between nodes seems to be broken in Debian 11 (bullseye) #3863

Comments

rgl commented Aug 15, 2021 • edited Loading

brandond commented Aug 17, 2021

rgl commented Aug 22, 2021

Nothing4You commented Aug 22, 2021

rgl commented Aug 22, 2021

rgl commented Aug 22, 2021 • edited Loading

brandond commented Aug 23, 2021

rgl commented Aug 23, 2021 • edited Loading

brandond commented Aug 24, 2021

rgl commented Aug 24, 2021

Nothing4You commented Aug 24, 2021

brandond commented Aug 26, 2021 • edited Loading

rgl commented Aug 26, 2021

rgl commented Aug 26, 2021 • edited Loading

jesseward commented Aug 27, 2021 • edited Loading

brandond commented Aug 27, 2021

Nothing4You commented Aug 31, 2021

ikaruswill commented Oct 1, 2021

nilbacardit26 commented Oct 4, 2021

brandond commented Oct 18, 2021 • edited Loading

homerzhou commented Feb 8, 2023

rgl commented Feb 9, 2023 • edited Loading

rgl commented Aug 15, 2021 •

edited

Loading

rgl commented Aug 22, 2021 •

edited

Loading

rgl commented Aug 23, 2021 •

edited

Loading

brandond commented Aug 26, 2021 •

edited

Loading

rgl commented Aug 26, 2021 •

edited

Loading

jesseward commented Aug 27, 2021 •

edited

Loading

brandond commented Oct 18, 2021 •

edited

Loading

rgl commented Feb 9, 2023 •

edited

Loading