[master] kube-proxy doesn't start up due to "apply caps: operation not permitted" error

[AkihiroSuda]

What happened:

The kube-proxy pod doesn't start up

What you expected to happen:
Should start.

How to reproduce it (as minimally and precisely as possible):

Set up

make && make install

(cd $GOPATH/src/k8s.io/kubernetes && git checkout v1.20.2)
docker build -t kindest/base:latest ./images/base
kind build node-image --base-image kindest/base:latest --type=bazel

kind create cluster --image kindest/node:latest

Result:

$ kubectl get pods -A
NAMESPACE            NAME                                         READY   STATUS             RESTARTS   AGE
kube-system          coredns-74ff55c5b-4w8n4                      0/1     Pending            0          39m
kube-system          coredns-74ff55c5b-q5vks                      0/1     Pending            0          39m
kube-system          etcd-kind-control-plane                      1/1     Running            0          39m
kube-system          kindnet-fqvxr                                1/1     Running            2          39m
kube-system          kube-apiserver-kind-control-plane            1/1     Running            0          39m
kube-system          kube-controller-manager-kind-control-plane   1/1     Running            0          39m
kube-system          kube-proxy-p72nb                             0/1     CrashLoopBackOff   6          39m
kube-system          kube-scheduler-kind-control-plane            1/1     Running            0          39m
local-path-storage   local-path-provisioner-78776bfc44-9wjrv      0/1     Pending            0          39m

$ kubectl get pods -n kube-system -o json kube-proxy-p72nb  | jq .status.containerStatuses
[
  {
    "containerID": "containerd://c9040c90a09b74ea044963a3ac5e57c26c951dacf21d7c77b0eed6ec6ab724bd",
    "image": "k8s.gcr.io/kube-proxy:v1.20.2",
    "imageID": "sha256:16fb33527f2df347f565f645eeb5dc20a371b7c7361c24eb20f9cb5ff3cb67f7",
    "lastState": {
      "terminated": {
        "containerID": "containerd://c9040c90a09b74ea044963a3ac5e57c26c951dacf21d7c77b0eed6ec6ab724bd",
        "exitCode": 128,
        "finishedAt": "2021-02-06T07:05:11Z",
        "message": "failed to create containerd task: OCI runtime create failed: container_linux.go:367: starting container process caused: apply caps: operation not permitted: unknown",
        "reason": "StartError",
        "startedAt": "1970-01-01T00:00:00Z"
      }
    },
    "name": "kube-proxy",
    "ready": false,
    "restartCount": 6,
    "started": false,
    "state": {
      "waiting": {
        "message": "back-off 5m0s restarting failed container=kube-proxy pod=kube-proxy-p72nb_kube-system(87e3cf9d-b3dc-4f5e-b3c3-d6af45d2a5a6)",
        "reason": "CrashLoopBackOff"
      }
    }
  }
]

Anything else we need to know?:

Possibly related to containerd/containerd#4717

Environment:

• kind version: (use kind version): acac774
• Kubernetes version: (use kubectl version): v1.20.2
• Docker version: (use docker info): moby/moby@3e0025e2fc
• OS (e.g. from /etc/os-release): Ubuntu 20.10, kernel 5.8.0-41-generic

[AkihiroSuda]

Maybe we can avoid hitting this issue by updating the parent Docker (and Podman) to support CAP_PERFMON CAP_BPF CAP_CHECKPOINT_RESTORE, but it will probably take a few years to get packaged for end users.

So, I guess we will need to write a wrapper script for runc to eliminate unsupported caps?
Or do we want to modify containerd to ignore unsupported caps?

[BenTheElder]

Hmm can't repro in our CI or my local dev. That's fun.

Or do we want to modify containerd to ignore unsupported caps?

I'm not sure if that's reasonable on the containerd side. We generally only ask for changes that have a legitimate use orthogonally to KIND.

In which case:

So, I guess we will need to write a wrapper script for runc to eliminate unsupported caps?

Seems likely.

[BenTheElder]

previously: #1906, moby/moby#41562
similar but crun / cri-o: cri-o/cri-o#4478

[BenTheElder]

seems we have a bit of a chicken and egg issue here 😞
moby/moby#41563

While these kernels support these capabilities, the current release of runc ships with an older version of gocapability/capability, and does not know about them, causing an error to be produced.

We can remove this restriction once opencontainers/runc@6dfbe9b is included in a runc release and once we stop supporting containerd 1.3.x (which ships with runc v1.0.0-rc92).

This will be fun for us, since as you said it will be some time before that reaches users.

[BenTheElder]

as for repro: my test hosts are not yet on kernel 5.8+ which your opencontainers/runc@6dfbe9b helpfully points out is when these were introduced.

[AkihiroSuda]

I'm not sure if that's reasonable on the containerd side. We generally only ask for changes that have a legitimate use orthogonally to KIND.

Aside from kind, minikube and k3d will need this fix.

I'm thinking of modifying GetAllCapabilities() to parse the CapEff field in /proc/self/status.
https://github.com/containerd/containerd/blob/v1.5.0-beta.1/oci/spec_opts.go#L784-L800

[aojea]

/cc

[thaJeztah]

related discussion in the runtime spec; opencontainers/runtime-spec#1071

so, for docker, this could be resolved once runc v1.0.0-rc93 is included in our containerd packages (must be tested with containerd 1.4.x), but would require the minimum "required" version in packaging to be raised (as older versions will no longer be compatible)

[AkihiroSuda]

PR: containerd/containerd#5017

[BenTheElder]

Thanks @thaJeztah! -- I think were going to have to work around this for some time though, we still see users with docker 18.X pretty often and it otherwise works fine, but worse we have the actual situation in kubernetes's CI like this:

[kubernetes node on GKE running containerd, as supplied by stable GKE for low maintenance ...
   (container in kubernetes pod running docker-in-containerd for building kubernetes etc., as supplied by an image I maintain
      (kind node container running in the kubernetes pod's docker, running containerd-in-docker, as supplied by KIND
           (pod container within the kind cluster.

turtles all the way down and all that ...

Even if we bump docker in CI to pickup this change ASAP, we still have the containerd on the real hosts. Right now CI is actually running on top of containerd 1.2.10 (!) for the nodes under wg-k8s-infra. There's definitely newer available (it's also still on Kubernetes 1.16.X ...) but I think it will be some time before the currently unreleased containerd 1.5.X makes it's way onto the CI nodes.

[dims]

/cc @dims

[BenTheElder]

This should be resolved in the base image now, which will be picked up in future node images and/or if the user is building and testing Kubernetes @ HEAD (if someone is not using kind @ HEAD for this particular purpose, please consider using a stable kind release instead!)

Thanks all!

[aojea]

@vrothberg I'm hitting this with this package

2021-12-17T02:08:47.9495856Z Get:5 https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_21.04  cri-o-runc 1.0.1~0 [3014 kB]

is it possible that runc version does not have the fix mentioned in #2058 (comment) ?

[vrothberg]

@aojea, the fix was included in run v1.0.0-rc93. cri-o-runc 1.0.1~0 is a more recent version so it should have the fix.

[aojea]

@aojea, the fix was included in run v1.0.0-rc93. cri-o-runc 1.0.1~0 is a more recent version so it should have the fix.

ok, then it was a red herring, thanks for the quick response, I'll keep digging

[thaJeztah]

@aojea perhaps related to situations as described in moby/moby#42906 (addressed by moby/moby#42933 on "master" in the Moby repo). The runc fix was to account for capabilities that were supported by the kernel, but not yet known to runc's code, but there were some other situations where (e.g. in a docker-in-docker setup) the capabilities were restricted in other ways. A similar "fix" is implemented for runc in opencontainers/runc#3240, but yet to be discussed.

[Chathuru]

@aojea, the fix was included in run v1.0.0-rc93. cri-o-runc 1.0.1~0 is a more recent version so it should have the fix.

ok, then it was a red herring, thanks for the quick response, I'll keep digging

I am using cri-o 1.22. Updating runc to 1.0.2-1 fixed the issues