Flannel CNI / DNS Issues with in-cluster Service Names when "hostNetwork: true"

[reefland]

Environmental Info:
K3s Version:

k3s version v1.25.4+k3s1 (0dc63334)
go version go1.19.3

Node(s) CPU architecture, OS, and Version:
Ubuntu 22.04.1 / Bare Metal

$ uname -a
Linux k3s01 5.15.0-58-generic #64-Ubuntu SMP Thu Jan 5 11:43:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:

$ sudo kubectl get nodes
NAME    STATUS   ROLES                       AGE    VERSION
k3s01   Ready    control-plane,etcd,master   309d   v1.25.4+k3s1
k3s02   Ready    control-plane,etcd,master   282d   v1.25.4+k3s1
k3s03   Ready    control-plane,etcd,master   289d   v1.25.4+k3s1
k3s04   Ready    worker                      55d    v1.25.4+k3s1
k3s05   Ready    worker                      55d    v1.25.4+k3s1

Describe the bug:
When I tried to enable hostNetwork: true for a pod, that pod is no longer able to resolve in-cluster DNS names. If I set both hostNetwork: true and dnsPolicy: ClusterFirstWithHostNet then neither internal nor external DNS names work.

With hostNetwork: false then both in-cluster and external DNS names work as expected.

Steps To Reproduce:
Using default flannel:

$ sudo  cat /var/lib/rancher/k3s/agent/etc/cni/net.d/10-flannel.conflist

{
  "name":"cbr0",
  "cniVersion":"1.0.0",
  "plugins":[
    {
      "type":"flannel",
      "delegate":{
        "hairpinMode":true,
        "forceAddress":true,
        "isDefaultGateway":true
      }
    },
    {
      "type":"portmap",
      "capabilities":{
        "portMappings":true
      }
    }
  ]
}

From Control-Plane Service:

ExecStartPre=/bin/sh -xc '! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service'
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/k3s \
    server \
        '--cluster-init' \
        '--token' \
        '[REDACTED]' \
        '--disable' \
        'traefik' \
        '--kube-apiserver-arg=feature-gates=MixedProtocolLBService=true' \
        '--kubelet-arg=image-gc-high-threshold=65' \
        '--kubelet-arg=image-gc-low-threshold=50' \
        '--disable' \
        'local-storage' \
        '--disable' \
        'servicelb' \
        '--tls-san=192.168.10.239' 

From Agent Service:

ExecStartPre=/bin/sh -xc '! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service'
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/k3s \
    agent \
        '--kubelet-arg=image-gc-low-threshold=50' \
        '--kubelet-arg=image-gc-high-threshold=65' 

Expected behavior:
I expected to be able to resolve in-cluster and external DNS names with hostNetwork enabled.

---

No Host Networking: In-Cluster & External DNS Work

$ kubectl run curl --image=radial/busyboxplus:curl --rm=true --stdin=true --tty=true --restart=Never 

[ root@curl:/ ]$ cat /etc/resolv.conf
search kube-system.svc.cluster.local svc.cluster.local cluster.local [REDACTED] localdomain
nameserver 10.43.0.10
options ndots:5

[ root@curl:/ ]$ curl http://grafana.monitoring.svc.cluster.local
<a href="/grafana/login">Found</a>.

[ root@curl:/ ]$ curl https://www.google.com | wc -l
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 14054    0 14054    0     0   164k      0 --:--:-- --:--:-- --:--:--  198k
16

Just hostNetwork: true: In-Cluster Fails but External DNS Works

This seems normal as the nameserver IP is the DNS for this LAN segment (for external DNS).

$ kubectl run curl --image=radial/busyboxplus:curl --rm=true --stdin=true --tty=true --restart=Never \
 --overrides='{"apiVersion": "v1", "spec": {"hostNetwork": true }}'
 
[ root@k3s05:/ ]$ cat /etc/resolv.conf
search [REDACTED] localdomain
nameserver 192.168.10.1

[ root@k3s05:/ ]$ curl http://grafana.monitoring.svc.cluster.local
curl: (6) Couldn't resolve host 'grafana.monitoring.svc.cluster.local'

[ root@k3s05:/ ]$ curl https://www.google.com | wc -l
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 14061    0 14061    0     0   190k      0 --:--:-- --:--:-- --:--:--  193k
16

Both hostNetwork: true and dnsPolicy: ClusterFirstWithHostNet: In-Cluster & External DNS Fail

$ kubectl run curl --image=radial/busyboxplus:curl --rm=true --stdin=true --tty=true --restart=Never \
  --overrides='{"apiVersion": "v1", "spec": {"hostNetwork": true, "dnsPolicy": "ClusterFirstWithHostNet"}}'

[ root@k3s05:/ ]$ cat /etc/resolv.conf
search kube-system.svc.cluster.local svc.cluster.local cluster.local [REDACTED] localdomain
nameserver 10.43.0.10
options ndots:5

[ root@k3s05:/ ]$ curl http://grafana.monitoring.svc.cluster.local
curl: (6) Couldn't resolve host 'grafana.monitoring.svc.cluster.local'

[ root@k3s05:/ ]$ curl https://www.google.com
curl: (6) Couldn't resolve host 'www.google.com'

While DNS fails, I can still curl the service directly and pod directly:

[ root@k3s05:/ ]$ curl http://10.43.174.7
<a href="/grafana/login">Found</a>.

[ root@k3s05:/ ]$ curl http://10.42.2.14:3000
<a href="/grafana/login">Found</a>.

---

Not clear if this is helpful:

$ for p in $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name); do kubectl logs --namespace=kube-system $p; done | uniq

[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
.:53
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[INFO] plugin/reload: Running configuration SHA512 = b941b080e5322f6519009bb49349462c7ddb6317425b0f6a83e5451175b720703949e3f3b454a24e77f3ffe57fd5e9c6130e528a5a1dd00d9000e4afd6c1108d
CoreDNS-1.9.4
linux/amd64, go1.19.1, 1f0a41a
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server

$ sudo k3s check-config

Verifying binaries in /var/lib/rancher/k3s/data/7c994f47fd344e1637da337b92c51433c255b387d207b30b3e0262779457afe4/bin:
- sha256sum: good
- links: aux/ip6tables should link to iptables-detect.sh (fail)
- links: aux/ip6tables-restore should link to iptables-detect.sh (fail)
- links: aux/ip6tables-save should link to iptables-detect.sh (fail)
- links: aux/iptables should link to iptables-detect.sh (fail)
- links: aux/iptables-restore should link to iptables-detect.sh (fail)
- links: aux/iptables-save should link to iptables-detect.sh (fail)

System:
- /var/lib/rancher/k3s/data/7c994f47fd344e1637da337b92c51433c255b387d207b30b3e0262779457afe4/bin/aux iptables v1.8.6 (legacy): ok
- swap: disabled
- routes: default CIDRs 10.42.0.0/16 or 10.43.0.0/16 already routed

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-58-generic
info: reading kernel config from /boot/config-5.15.0-58-generic ...

Generally Necessary:
- cgroup hierarchy: cgroups V2 mounted, cpu|cpuset|memory controllers status: good
- /usr/sbin/apparmor_parser
apparmor: enabled and tools installed
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_POSIX_MQUEUE: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: enabled (as module)
- CONFIG_CGROUP_NET_PRIO: enabled
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: missing
- CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module)
- CONFIG_IP_SET: enabled (as module)
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: enabled
- CONFIG_IP_VS_PROTO_UDP: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled (as module)
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: enabled
      - CONFIG_CRYPTO_SEQIV: enabled
      - CONFIG_CRYPTO_GHASH: enabled
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled (as module)
      - CONFIG_XFRM_ALGO: enabled (as module)
      - CONFIG_INET_ESP: enabled (as module)
      - CONFIG_INET_XFRM_MODE_TRANSPORT: missing
- Storage Drivers:
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled (as module)

STATUS: 6 (fail)

[brandond]

cc @rbrtbnfgl @thomasferrandiz @manuelbuil

[reefland]

I just upgraded to latest available on 1.25, and have better results:

$ k get nodes                                                                                                                                              127 ↵
NAME    STATUS   ROLES                       AGE    VERSION
k3s01   Ready    control-plane,etcd,master   312d   v1.25.6+k3s1
k3s02   Ready    control-plane,etcd,master   286d   v1.25.6+k3s1
k3s03   Ready    control-plane,etcd,master   293d   v1.25.6+k3s1
k3s04   Ready    worker                      58d    v1.25.6+k3s1
k3s05   Ready    worker                      58d    v1.25.6+k3s1

I can now curl internal service:

$ kubectl run curl --image=radial/busyboxplus:curl --rm=true --stdin=true --tty=true --restart=Never \
  --overrides='{"apiVersion": "v1", "spec": {"hostNetwork": true, "dnsPolicy": "ClusterFirstWithHostNet"}}'
  
[ root@k3s04:/ ]$ curl http://grafana.monitoring.svc.cluster.local
<a href="/grafana/login">Found</a>.

No change to /etc/resolv.conf:

[ root@k3s04:/ ]$ cat /etc/resolv.conf
search kube-system.svc.cluster.local svc.cluster.local cluster.local [REDACTED] localdomain
nameserver 10.43.0.10
options ndots:5

I didn't see anything obvious within the release notes that would explain this. I do have a single node test cluster on the side I can still reproduce the original issue as it is still on v1.25.4+k3s1 if that would be helpful.

[rbrtbnfgl]

@brandond could it be related to the containerd bug?

[baowj-678]

I also encountered this problem. Only pods on the node that runing coredns-xxxx can visited In-Cluster DNS.
My k3s Cluster is:
k3s version v1.24.8+k3s1 (648004e4)
go version go1.18.8

[brandond]

@baowj-678 that usually indicates a problem with your kernel dropping vxlan packets between nodes. This is caused by a bug in the kernel checksum offload code. See the ethtool command at flannel-io/flannel#1279

[rbrtbnfgl]

That issue was solved from flannel v0.20.2. K3s v1.24.8 it's using v0.20.1 and also v1.25.4 uses that version then both issues could be related to that.

[baowj-678]

Thanks, I solved my problem by following this issue flannel-io/flannel#1279.

[reefland]

Thanks all, looks like upgrade of K3S updated the embedded flannel which resolved my issue. Closing this.