Context Deadline Exceeded (CreateContainerError) warnings on pods with attached PVCs

[andrew-wilson-88]

I have a number of Cronjobs running at 1 minute intervals with attached PVC storage (EBS GP3 volumes). Occasionally we see jobs spinning up and being delayed for varying times reporting "Context Deadline Exceeded". The pods do eventually come online after this time and complete successfully. Events from one such pod look like the following:

Events:
  Type     Reason          Age                 From               Message
  ----     ------          ----                ----               -------
  Normal   Scheduled       23m                 default-scheduler  Successfully assigned sample-namespace/sample-cron-28285183-tzmt2 to ip-10-0-169-183.ap-southeast-2.compute.internal
  Normal   AddedInterface  23m                 multus             Add eth0 [10.131.0.68/23] from ovn-kubernetes
  Warning  Failed          11m                 kubelet            Error: kubelet may be retrying requests that are timing out in CRI-O due to system load. Currently at stage container volume configuration: the requested container k8s_sample-cron_sample-cron-28285183-tzmt2_sample-namespace_51cd9f1d-7a54-4210-90aa-2e4111755480_2 is now ready and will be provided to the kubelet on next retry: error reserving ctr name k8s_sample-cron_sample-cron-28285183-tzmt2_sample-namespace_51cd9f1d-7a54-4210-90aa-2e4111755480_2 for id 59ef5eec8345e9bbb573fd6300d8d8c89e9f614664125fecf339d55c59147887: name is reserved
  Normal   Pulled          7m3s (x9 over 23m)  kubelet            Container image "image-registry.openshift-image-registry.svc:5000/openshift/cron-container" already present on machine
  Warning  Failed          7m3s (x7 over 21m)  kubelet            Error: context deadline exceeded
  Normal   Created         6m                  kubelet            Created container sample-cron
  Normal   Started         6m                  kubelet            Started container sample-cron

I'm trying to gain some further insight into the processes running when setting up the context so that I can find a way to reduce the amount of time it takes for the cron to start.

To me it looks to be similar to cri-o/cri-o#6185 - however that issue refers to object storage rather than a block device, is it possible that SElinux labelling could be the issue, and is there a way in OKD to test with the "TrySkipVolumeSELinuxLabel" label?

Regards,
Andrew

[vrutkovs]

is it possible that SElinux labelling could be the issue

It looks very similar, so I think its the same issue

is there a way in OKD to test with the "TrySkipVolumeSELinuxLabel" label?

Yes. You should create a MachineConfig with CRIO config change, see openshift/enhancements#362 (comment) for example

[andrew-wilson-88]

Thanks for the quick reply and the example, I'll trial it out and see if there is any noticeable improvement.

Regards,
Andrew

[andrew-wilson-88]

I ended up testing this on a isolated single node and the pod startup time was vastly improved when attaching a well populated PVC volume. I've since rolled this out to all nodes and we noticed the Load Average on worker nodes noticeably drop with no ill-effects.