Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

failed to create bridge "cni0": could not add "cni0": operation not supported #370

Closed
ChenQi1989 opened this issue Aug 13, 2019 · 17 comments
Closed

failed to create bridge "cni0": could not add "cni0": operation not supported #370

ChenQi1989 opened this issue Aug 13, 2019 · 17 comments

Comments

@ChenQi1989
Copy link

I'm trying to setup k8s using flannel. But I'm blocked by this error. Hope someone could give me some suggestion.

I used the following commands:

  1. swapoff -a
  2. kubeadm init --pod-network-cidr=10.244.0.0/16
  3. sysctl net.bridge.bridge-nf-call-iptables=1
  4. mkdir -p $HOME/.kube
    cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    chown $(id -u):$(id -g) $HOME/.kube/config
  5. kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

I also added some output in kubelet and link_linux.go to help debugging. But I'm now blocked here. Below are the related logs. Any idea?

Aug 13 03:30:39 yocto-k8s-master kubelet[9968]: I0813 03:30:39.018857 9968 raw_exec.go:40] chenqi: raw_exec.ExecPlugin &{/opt/cni/bin/flannel [/opt/cni/bin/flannel] [PATH=/usr/local/sbin:/usr/local/bin:/usr
/sbin:/usr/bin:/sbin:/bin INVOCATION_ID=ed7608a7f620418695c029825e3d374d CNI_ARGS=IgnoreUnknown=1;K8S_POD_NAMESPACE=kube-system;K8S_POD_NAME=coredns-5f7fc64c95-x4l8t;K8S_POD_INFRA_CONTAINER_ID=ce1302be7aedaf89
32763b461a287fa5437a3889580fa615c7422f43f65d4c54 KUBELET_KUBEADM_ARGS=--cgroup-driver=systemd --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1 --resolv-conf=/run/systemd/resolve/resolv.con
f CNI_IFNAME=eth0 LANG=C JOURNAL_STREAM=8:3119020 KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf CNI_CONTAINERID=ce1302be7aedaf8
932763b461a287fa5437a3889580fa615c7422f43f65d4c54 CNI_NETNS=/proc/12016/ns/net KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml KUBELET_EXTRA_ARGS=-v 4 CNI_COMMAND=ADD CNI_PATH=/opt/cni/bin] {"cniVer
sion":"","delegate":{"hairpinMode":true,"isDefaultGateway":true},"name":"cbr0","type":"flannel"} %!s(*os.File=&{0xc00009e120}) [] %!s(*syscall.SysProcAttr=) %!s(*os.Process=) %!s(*context.empt
yCtx=0xc000052038) %!s(bool=false) [] [] [] [] %!s(chan error=) %!s(chan struct {}=)}
Aug 13 03:30:39 yocto-k8s-master kubelet[9968]: 2019/08/13 03:30:39 chenqi: link_linux.go: link type is *netlink.Bridge
Aug 13 03:30:39 yocto-k8s-master kubelet[9968]: 2019/08/13 03:30:39 chenqi: link_linux.go: after req.AddData
Aug 13 03:30:39 yocto-k8s-master systemd[1]: Created slice libcontainer_12120_systemd_test_default.slice.
Aug 13 03:30:39 yocto-k8s-master kubelet[9968]: I0813 03:30:39.030635 9968 factory.go:177] Factory "docker" was unable to handle container "/libcontainer_12120_systemd_test_default.slice"
Aug 13 03:30:39 yocto-k8s-master kubelet[9968]: I0813 03:30:39.030661 9968 factory.go:166] Error trying to work out if we can handle /libcontainer_12120_systemd_test_default.slice: /libcontainer_12120_syste
md_test_default.slice not handled by systemd handler
Aug 13 03:30:39 yocto-k8s-master kubelet[9968]: I0813 03:30:39.030668 9968 factory.go:177] Factory "systemd" was unable to handle container "/libcontainer_12120_systemd_test_default.slice"
Aug 13 03:30:39 yocto-k8s-master kubelet[9968]: I0813 03:30:39.030678 9968 factory.go:170] Factory "raw" can handle container "/libcontainer_12120_systemd_test_default.slice", but ignoring.
Aug 13 03:30:39 yocto-k8s-master kubelet[9968]: I0813 03:30:39.030689 9968 manager.go:908] ignoring container "/libcontainer_12120_systemd_test_default.slice"
Aug 13 03:30:39 yocto-k8s-master systemd[1]: Removed slice libcontainer_12120_systemd_test_default.slice.
Aug 13 03:30:39 yocto-k8s-master systemd-udevd[12091]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Aug 13 03:30:39 yocto-k8s-master systemd-udevd[12091]: link_config: could not get ethtool features for cni0
Aug 13 03:30:39 yocto-k8s-master systemd-udevd[12091]: Could not set offload features of cni0: No such device
Aug 13 03:30:39 yocto-k8s-master kubelet[9968]: 2019/08/13 03:30:39 chenqi: link_linux.go: req.Execute(unix.NETLINK_ROUTE, 0) failed
Aug 13 03:30:39 yocto-k8s-master kubelet[9968]: I0813 03:30:39.041036 9968 raw_exec.go:51] chenqi: in pluginErr: error: exit status 1; output = {
Aug 13 03:30:39 yocto-k8s-master kubelet[9968]: "code": 100,
Aug 13 03:30:39 yocto-k8s-master kubelet[9968]: "msg": "failed to create bridge "cni0": could not add "cni0": operation not supported"
Aug 13 03:30:39 yocto-k8s-master kubelet[9968]: }
Aug 13 03:30:39 yocto-k8s-master kubelet[9968]: I0813 03:30:39.041077 9968 raw_exec.go:59] chenqi: pluginErr returning error message: failed to create bridge "cni0": could not add "cni0": operation not supported
Aug 13 03:30:39 yocto-k8s-master kubelet[9968]: E0813 03:30:39.041085 9968 cni.go:338] Error adding kube-system_coredns-5f7fc64c95-x4l8t/ce1302be7aedaf8932763b461a287fa5437a3889580fa615c7422f43f65d4c54 to network flannel/cbr0: failed to create bridge "cni0": could not add "cni0": operation not supported
@ChenQi1989
Copy link
Author

This seems to be a regression. As if I use v0.7.5, there's no such problem.

commit a62711a (tag: v0.7.5, origin/v0.7)

@ChenQi1989
Copy link
Author

I've tracked down to the following commit. I think it caused such regression.

commit dc53699
Author: Sebastian Sch [email protected]
Date: Sun Nov 18 15:43:35 2018 +0200

vendor folder bump.

@squeed
Copy link
Member

squeed commented Aug 13, 2019

Interesting
The latest release. v0.7.6, should fix this.

@ChenQi1989
Copy link
Author

@squeed v0.7.5 works for me. Checking the v0.7 branch, I think v0.7.6 should also work.
The problem I encountered is about origin/master branch.

@dcbw
Copy link
Member

dcbw commented Aug 21, 2019

@ChenQi1989 git master has the same commit as v0.7.6 though...

@ChenQi1989
Copy link
Author

@dcbw I mean things are working well on v0.7 branch, including v0.7.5 and v0.7.6. But they are working well on master branch. I tracked down to the following commit.
"""
commit dc53699
Author: Sebastian Sch [email protected]
Date: Sun Nov 18 15:43:35 2018 +0200

vendor folder bump.

"""

But I don't know the root cause.
I tried k8s 1.15.1 + cni plugin master + flannel setup, the problem could be reproduced.

@ChenQi1989
Copy link
Author

@dcbw Sorry for typo above. I mean the master branch is NOTworking well.

@Promaethius
Copy link

Promaethius commented Aug 22, 2019 •
edited
Loading

Having the same issue on master and v0.8.2
Update: I have fixed the issue!
Backporting my cni and net.d to 0.7.6 and 3.0, it would output that it couldn't create veth devices. Rebuilt my kernel with CONFIG_VETH and 0.7.6, 3.0 worked in building the cni0 bridge.
Switched to master branch and rebuilt the cni plugins and used 4.0 cni and it works!

UPDATE: I did not fix the issue.
if the 7.6 cni plugins are run before the 8.x plugins, then the 8.x plugins work fine. However, after a fresh reboot I try to run 8.x, it shows the same error.

@squeed
Copy link
Member

squeed commented Aug 28, 2019

I suspect I know the issue.

Can you try building the binaries manually with this line commented out? https://github.com/containernetworking/plugins/blob/master/plugins/main/bridge/bridge.go#L224

@tgross tgross mentioned this issue Nov 4, 2019
Merged
@carlosedp
Copy link
Contributor

I'm seeing this behavior on a Raspberry Pi with plugins v0.8.2.

When I reverted to v0.7.5 it worked.

pi@raspberrypi:~/faasd $ uname -a
Linux raspberrypi 4.19.75-v7+ #1270 SMP Tue Sep 24 18:45:11 BST 2019 armv7l GNU/Linux
pi@raspberrypi:~/faasd $ cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

cc. @alexellis

@carlosedp carlosedp mentioned this issue Jan 9, 2020
Closed
@carlosedp
Copy link
Contributor

Confirming that commenting the line as pointed by @squeed fixes the problem and the bridge interface is created.

@alexellis
Copy link

Does git blame show why it was added and how it was tested in the commit message? I'd be curious to know if this is an issue that affects x86_64, or just RPi/armhf.

@alexellis
Copy link

@bboreham cc

@carlosedp
Copy link
Contributor

Maybe @SchSeba could also help out figuring if it's something missing on Kernel side.

@carlosedp carlosedp mentioned this issue Jan 9, 2020
Merged
@l1key0u
Copy link

l1key0u commented Sep 17, 2020

I try to reinstall k8s, it is work!
Maybe lost somethings.

@l1key0u
Copy link

l1key0u commented Sep 17, 2020

I try to reinstall k8s, it is work!
Maybe lost somethings.

If the node is not running coredns, cni0 will not be created.

@bboreham
Copy link
Contributor

Closed by #434

If the node is not running coredns, cni0 will not be created.

@l1key0u sounds unlikely. If you're still having problems with the Flannel CNI plugin please open a new issue and supply full details.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@carlosedp @squeed @Promaethius @alexellis @bboreham @l1key0u @dcbw @ChenQi1989