pod cycle restart,Pod sandbox changed, it will be killed and re-created.

[pruidong]

Summary

I installed MicroK8s v1.30.4 revision 7167 on Ubuntu 24.04. Using microK8s status to check the status, MicroK8s is running, but when checking the status of pods, only the Calico node is running, and all other pods are ContainerCreating The ContainerCreating status has been going on for a long time. I have checked the detailed logs and the error reason is: Pod sandbox changed, it will be killed and re created. I used #1775 The method in the system involves stopping snap.microk8s.daemon-apiserver picker to stop the apiserver picker service, but the pod still restarts repeatedly. What should I do? thank you.

System:

Ubuntu 24.04

MicroK8s Version:

1.30.4

What Should Happen Instead?

All pods are in running state

Reproduction Steps

1. sudo snap install microk8s --classic
2. microk8s status --wait-ready
3. microk8s enable dashboard
4. microk8s enable dns
5. microk8s kubectl get all --all-namespaces
6. microk8s kubectl get pods --all-namespaces

Detail

> microk8s kubectl get pods --all-namespaces
NAMESPACE     NAME                                         READY   STATUS              RESTARTS      AGE
kube-system   calico-kube-controllers-796fb75cc-8h6m2      0/1     ContainerCreating   0             17h
kube-system   calico-node-cf48j                            1/1     Running             1 (16h ago)   17h
kube-system   coredns-5986966c54-65tdp                     0/1     ContainerCreating   0             17h
kube-system   dashboard-metrics-scraper-795895d745-72jrs   0/1     ContainerCreating   0             17h
kube-system   hostpath-provisioner-7c8bdf94b8-lgszw        0/1     ContainerCreating   0             17h
kube-system   kubernetes-dashboard-6796797fb5-bdrsj        0/1     ContainerCreating   0             17h
kube-system   metrics-server-7cff7889bd-hx2q6              0/1     ContainerCreating   0             17h

> microk8s version
MicroK8s v1.30.4 revision 7167

> microk8s kubectl describe pods kubernetes-dashboard-6796797fb5-bdrsj -n kube-system
Name:             kubernetes-dashboard-6796797fb5-bdrsj
Namespace:        kube-system
Priority:         0
Service Account:  kubernetes-dashboard
Node:             xxxx-pc/192.168.254.131
Start Time:       Tue, 03 Sep 2024 22:17:25 +0800
Labels:           k8s-app=kubernetes-dashboard
                  pod-template-hash=6796797fb5
Annotations:      <none>
Status:           Pending
SeccompProfile:   RuntimeDefault
IP:               
IPs:              <none>
Controlled By:    ReplicaSet/kubernetes-dashboard-6796797fb5
Containers:
  kubernetes-dashboard:
    Container ID:  
    Image:         kubernetesui/dashboard:v2.7.0
    Image ID:      
    Port:          8443/TCP
    Host Port:     0/TCP
    Args:
      --auto-generate-certificates
      --namespace=kube-system
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Liveness:       http-get https://:8443/ delay=30s timeout=30s period=10s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /certs from kubernetes-dashboard-certs (rw)
      /tmp from tmp-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2gs6r (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   False 
  Initialized                 True 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:
  kubernetes-dashboard-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kubernetes-dashboard-certs
    Optional:    false
  tmp-volume:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-2gs6r:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node-role.kubernetes.io/master:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason          Age                  From     Message
  ----    ------          ----                 ----     -------
  Normal  SandboxChanged  63s (x224 over 16h)  kubelet  Pod sandbox changed, it will be killed and re-created.

> microk8s inspect

Inspecting system
Inspecting Certificates
Inspecting services
Service snap.microk8s.daemon-cluster-agent is running
Service snap.microk8s.daemon-containerd is running
Service snap.microk8s.daemon-kubelite is running
Service snap.microk8s.daemon-k8s-dqlite is running
FAIL:  Service snap.microk8s.daemon-apiserver-kicker is not running
For more details look at: sudo journalctl -u snap.microk8s.daemon-apiserver-kicker
Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
Copy processes list to the final report tarball
Copy disk usage information to the final report tarball
Copy memory usage information to the final report tarball
Copy server uptime to the final report tarball
Copy openSSL information to the final report tarball
Copy snap list to the final report tarball
Copy VM name (or none) to the final report tarball
Copy current linux distribution to the final report tarball
Copy asnycio usage and limits to the final report tarball
Copy inotify max_user_instances and max_user_watches to the final report tarball
Copy network configuration to the final report tarball
Inspecting kubernetes cluster
Inspect kubernetes cluster
Inspecting dqlite
Inspect dqlite
cp: cannot stat '/var/snap/microk8s/7167/var/kubernetes/backend/localnode.yaml': No such file or directory

Building the report tarball
Report tarball is at /var/snap/microk8s/7167/inspection-report-20240904_155434.tar.gz

[ktsakalozos]

Hi @pruidong could you please attach the tarball created by the inspection script so we can see the logs of the cluster? Thank you.