Some deployed cni plugins are not statically compiled

[nitrotm]

Describe the bug
On an alpine-based native server (which is itself based on busybox), /opt/cni/bin/antrea cannot execute due to being dynamically linked to some system libraries. This is not the case for some other standard plugins, such as loopback.

To Reproduce
Try to run antrea cni plugin binary on a busybox system or alternatively with incompatible libpthread or libc. In particulary, it didn't work on k3os.

Expected
As with other cni plugins, it should be statically compiled so that Antrea can be deployed on any system.

Actual behavior
Antrea cannot be deployed successfully (spawning non-host containers fails when network plugin is calling /opt/cni/bin/antrea).

Versions:

• Antrea version: 1.0.0
• Kubernetes version: 1.20
• Container runtime: containerd
• Linux kernel version: 5.4.0

Additional context
None.

[tnqn]

Hi @nitrotm, thanks for the report! Making antrea cni binary statically compiled makes sense to me. It applies to other binaries that will run out of container too, e.g. antctl, antrea-octant-plugin.
Or maybe we could just compile all of the binaries statically if no real drawbacks. What do you think? @jianjuns @antoninbas

@nitrotm would you be willing to create a PR for this?

[tnqn]

And to confirm, I assume the issue could cause the container runtime fail to call antrea cni, but it shouldn't fail on initContainer of antrea-agent while installing cni. Could you paste the failure of initContainer if there is? I couldn't reproduce it with k3os.

[jianjuns]

Agreed on compiling Antrea CNI statically.

[nitrotm]

@tnqn sorry, that's right, not related to initContainer. I was trying a whole bunch of different cni solutions (calico, cilium, kube-ovn, etc) and forgot the exact root cause of the error. I did the deploy from scratch again and here is more detailed info:

If you deploy antrea and start a simple pod (eg. busybox), then you'll see the following in the events:

│   Warning  FailedCreatePodSandBox  48s   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "aa771c57bc881ace3ec1548653ca27f7382940d4c312893576a5432d5b46af9d": netplugin failed with no error message: fork/exec /opt/cni/bin/antrea: no such file or d │

But the file is there on the host /opt/cni/bin/antrea, simply not executable, and ldd reports:

$> ldd /opt/cni/bin/antrea 
        /lib64/ld-linux-x86-64.so.2 (0x7f30d161b000)
        libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x7f30d161b000)
        libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f30d161b000)

which indicates some kind of library issue (eg. it looks like libc/libpthread linkage are impossible to satisfy in this busybox setup).

[nitrotm]

As per the question about creating a PR, I am probably not the right person for doing this right now because I feel I don't know yet enough about CNI internals and practically nothing about the way antrea is built and packaged.

I'll give antrea a try when this is solved and eventually will replace calico in my self-provisioned k8s clusters, depending upon how well it will fit my use-case. In the case a migration to antrea makes sense, I would be more keen on contributing to the project in the future.

[antoninbas]

Thanks for reporting this @nitrotm. I can look into this before the next release (Antrea v1.1). Will assign this to me, but if someone wants to start working on this, please comment here and I can assign the issue to you.