weave-kube crash: ipset v7.2: Set cannot be destroyed: it is in use by a kernel component

[drigz]

What happened?

weave-kube is crashing with:

$ kubectl -n kube-system logs weave-net-x4grk weave
ipset v7.2: Set cannot be destroyed: it is in use by a kernel component

How to reproduce it?

I installed Weave with nftables on minikube + kubernetes 1.17, IIRC it initially appeared to work but Weave started failing. If I switch to iptables-legacy, the issue goes away, but returns if I switch to nftables, reboot, and recreate the cluster.

sudo MINIKUBE_HOME=$HOME minikube delete
sudo update-alternatives --set iptables /usr/sbin/iptables-nft
reboot
( cd /tmp \                                                   
  && curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube_1.7.3-0_amd64.deb \
  && sudo dpkg -i minikube_1.7.3-0_amd64.deb )
sudo mkdir -p /etc/kubernetes
echo "nameserver 8.8.8.8" | sudo tee /etc/kubernetes/resolv.conf
sudo MINIKUBE_HOME=$HOME CHANGE_MINIKUBE_NONE_USER=true KUBECONFIG=$HOME/.kube/config \
  minikube start --vm-driver=none \
    --extra-config kubelet.resolv-conf=/etc/kubernetes/resolv.conf \
    --extra-config kubelet.runtime-cgroups=/lib/systemd/system/kubelet.service \
    --extra-config kubelet.kubelet-cgroups=/lib/systemd/system/kubelet.service \
    --extra-config=kubelet.eviction-hard="memory.available<300Mi,nodefs.available<600Mi,imagefs.available<600Mi" \
    --apiserver-ips 127.0.0.1 --apiserver-name localhost \
    --network-plugin=cni --extra-config=kubelet.network-plugin=cni

Then install weave with the following settings:

                - name: IPALLOC_RANGE
                  value: 192.168.9.0/24
                - name: IPTABLES_BACKEND
                  value: nft
                # Prevent external connections to the datapath port.
                - name: EXTRA_ARGS
                  value: --host=127.0.0.1

Anything else we need to know?

Debugging steps tried to no avail:

• rebooting
• sudo ipset list -o save | cut -d' ' -f2 | sudo xargs -n 1 ipset destroy

This seems similar to #2915, but the crash comes from weave-kube rather than weave-npc.

I plan to try reimaging the machine, let me know if there's anything I should do to debug first.

Versions:

$ weave version
weave script 2.6.4
weave 2.6.4
$ docker version
19.03
$ uname -a
Debian Buster-like, based on 5.5.17
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:52:00Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:07:13Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}
$ ipset version
ipset v6.38, protocol version: 6
# kernel protocol version is 7

Logs:

$ kubectl -n kube-system logs weave-net-x4grk weave
ipset v7.2: Set cannot be destroyed: it is in use by a kernel component

Network:

$ ip route
default via 192.168.0.1 dev ens4 proto dhcp metric 100 
192.168.0.1 dev ens4 proto dhcp scope link metric 100 
192.168.10.0/24 dev docker0 proto kernel scope link src 192.168.10.1 linkdown
$ ip -4 -o addr
1: lo    inet 127.0.0.1/8 scope host lo\       valid_lft forever preferred_lft forever
2: ens4    inet 192.168.7.51/32 scope global dynamic noprefixroute ens4\       valid_lft 3242sec preferred_lft 3242sec
3: docker0    inet 192.168.10.1/24 brd 192.168.10.255 scope global docker0\       valid_lft forever preferred_lft forever
$ sudo iptables-save
https://gist.github.com/drigz/95b9dd7374034555d9817bf6f1866e79

[bboreham]

Thanks for the detailed report.

Most of the places that weave tries to destroy an ipset it ignores the error; the only one I can see that doesn't ignore is

weave/prog/weave-kube/launch.sh

Line 51 in 11d75b2

ipset destroy weave-kube-test

. This also matches the symptom that you get exactly one line in the log.

Could you print out the list of all ipsets after it has run, failed, and printed the error?

[drigz]

weave-kube-test is indeed there:

$ sudo ipset list -o save
create weave-P.B|!ZhkAr5q=XZ?3}tMBA+0 hash:ip family inet hashsize 1024 maxelem 65536 comment
create weave-E1ney4o[ojNrLk.6rOHi;7MPE hash:ip family inet hashsize 1024 maxelem 65536 comment
create weave-iuZcey(5DeXbzgRFs8Szo]+@p hash:ip family inet hashsize 1024 maxelem 65536 comment
create weave-Rzff}h:=]JaaJl/G;(XJpGjZ[ hash:ip family inet hashsize 1024 maxelem 65536 comment
create weave-41s)5vQ^o/xWGz6a20N:~?#|E hash:ip family inet hashsize 1024 maxelem 65536 comment
create weave-4vtqMI+kx/2]jD%_c0S%thO%V hash:ip family inet hashsize 1024 maxelem 65536 comment
create weave-]B*(W?)t*z5O17G044[gUo#$l hash:ip family inet hashsize 1024 maxelem 65536 comment
create weave-sui%__gZ}{kX~oZgI_Ttqp=Dp hash:ip family inet hashsize 1024 maxelem 65536 comment
create weave-mF}1zpEo4W6iYroE^=:V3{S6W hash:ip family inet hashsize 1024 maxelem 65536 comment
create weave-;rGqyMIl1HN^cfDki~Z$3]6!N hash:ip family inet hashsize 1024 maxelem 65536 comment
create weave-s_+ChJId4Uy_$}G;WdH|~TK)I hash:ip family inet hashsize 1024 maxelem 65536 comment
create weave-k?Z;25^M}|1s7P3|H9i;*;MhG hash:ip family inet hashsize 1024 maxelem 65536 comment
create weave-kube-test hash:ip family inet hashsize 1024 maxelem 65536

If I run sudo ipset destroy weave-kube-test it destroy it successfully, but it is recreated by weave the next time it starts, and weave continues to fail.

[bboreham]

OK, so somehow when you run the destroy command it works but when weave-kube runs the same command it fails.

Two hypotheses: (a) the difference is in the binary (b) the difference is in the timing.

Could you try running the sequence of 9 commands from the launch script as a script on the host, to see if it ever fails that way? And if it fails, can you still succeed with the single ipset destroy ?

You said switching to iptables-legacy makes a difference - does that change the above? Does it change the ipset binary?

You can repeat the experiment inside the Weave container:

docker run --privileged -ti --entrypoint=/bin/sh weaveworks/weave-kube:2.6.4

[drigz]

The difference appears to be timing. I tried a few times on the host and normally the first ipset destroy fails, but a second try "immediately" after succeeds. (sometimes the second fails too)

$ sudo bash -x
# xt_set_exists; ipset destroy weave-kube-test
+ xt_set_exists
+ iptables -F WEAVE-KUBE-TEST
+ true
+ iptables -X WEAVE-KUBE-TEST
+ true
+ ipset destroy weave-kube-test
+ ipset create weave-kube-test hash:ip
+ iptables -t filter -N WEAVE-KUBE-TEST
+ iptables -A WEAVE-KUBE-TEST -m set --match-set weave-kube-test src -j DROP
+ iptables -F WEAVE-KUBE-TEST
+ iptables -X WEAVE-KUBE-TEST
+ ipset destroy weave-kube-test
ipset v6.38: Kernel support protocol versions 6-7 while userspace supports protocol versions 6-6
Set cannot be destroyed: it is in use by a kernel component
+ '[' -z '' ']'
+ ipset destroy weave-kube-test

I get no error when running with docker run. I also tried adding --network=host but it still succeeds. No idea why that would behave differently to both the host & cluster.

[bboreham]

Great, thanks for helping drill into this.

I wish I knew more about how this all works inside the kernel.

Maybe we can extend the launch script to retry the destroy a couple of times, with a short delay.

[drigz]

https://www.reddit.com/r/linuxquestions/comments/gnwrk9/new_race_condition_in_iptables/ looks similar.

I found some threads refering to ipset flush (although I can't work out what "flush" means in this context and the docs just say it flushes the set 🙄). I tried adding ipset flush before ipset destroy but it had no effect.

[drigz]

I've hacked the YAML to work around this. So far it's working OK, but I'll let you know if I run into any other issues.

              command:
                - /bin/sh
                - -c
                - sed '/ipset destroy weave-kube-test$/ i sleep 1' /home/weave/launch.sh | /bin/sh

[bodanc]

hey everyone;
with the 4.18.0-193.el8.x86_64 version of the centOS 8 kernel (base) we don't run into this issue while we were able to reproduce it on the 4.18.0-193.14.2.el8_2.x86_64 build of centOS 8;
basically, if we do sudo yum update, we can repro. it
hth;

i also noticed there's another issue opened for the same problem:
#3847

[kavyabgowda]

The issue still persists. Any working solution?

[drigz]

@bboreham Should we merge the "sleep" workaround if we don't have an idea how to solve it? It has been surprisingly stable for me, I don't think I've seen the error since adding the sleep.

[kavyabgowda]

I'm still getting the same error with Centos 8 worker node and "Ubuntu 20.04.1 LTS" master node. If I add sleep then only one weave pod on both nodes are coming up. If I use latest weave yml then weave network is meeting the desired state on master node but on worker centos 8 node it's crashing with "ipset v7.2: Set cannot be destroyed: it is in use by a kernel component"

[kavyabgowda]

It's working now. I tried by adding sleep, but didn't work. So I reverted my changes and left. After so many restarts it's working. I'm not sure what was the issue and how it fixed.

[bboreham]

Closed by #3882