coredns pod not in ready state

[shrsam]

Hello,
When trying to bring up the kubernetes pods during RIC Kubernetes Cluster Installation, I am facing an issue where pods related to coreDNS are running but not in the ready state. After checking the logs related to CoreDNS, I can see that there is a taint which is not tolerated which is stopping the coreDNS from moving to ready state.
I have tried to debug this but unable to resolve the issue successfully. Could you please check the issue and let me know how to make sure that the CoreDNS pods are up and ready.
Attached the relevant log files.

aerpawops@aerpaw17:~$ sudo kubectl get pods --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system coredns-5644d7b6d9-twr2q 0/1 Running 1 7m52s 10.244.0.7 aerpaw17
kube-system coredns-5644d7b6d9-wrlvh 0/1 Running 1 7m52s 10.244.0.6 aerpaw17
kube-system etcd-aerpaw17 1/1 Running 1 7m9s 10.153.13.85 aerpaw17
kube-system kube-apiserver-aerpaw17 1/1 Running 1 7m1s 10.153.13.85 aerpaw17
kube-system kube-controller-manager-aerpaw17 1/1 Running 1 6m45s 10.153.13.85 aerpaw17
kube-system kube-flannel-ds-wdrcx 1/1 Running 1 7m52s 10.153.13.85 aerpaw17
kube-system kube-proxy-9jqm9 1/1 Running 1 7m52s 10.153.13.85 aerpaw17
kube-system kube-scheduler-aerpaw17 1/1 Running 1 7m8s 10.153.13.85 aerpaw17
kube-system tiller-deploy-7d7bc87bb-wfsdn 1/1 Running 1 6m39s 10.244.0.5 aerpaw17

aerpawops@aerpaw17:~$ sudo kubectl describe pod/coredns-5644d7b6d9-twr2q -n kube-system

Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True

Events:
Type Reason Age From Message

---

Warning FailedScheduling default-scheduler 0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.
Warning FailedScheduling default-scheduler 0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.
Normal Scheduled default-scheduler Successfully assigned kube-system/coredns-5644d7b6d9-twr2q to aerpaw17
Normal Pulled 10m kubelet, aerpaw17 Container image "k8s.gcr.io/coredns:1.6.2" already present on machine
Normal Created 10m kubelet, aerpaw17 Created container coredns
Normal Started 10m kubelet, aerpaw17 Started container coredns
Warning Unhealthy 9m6s (x6 over 9m56s) kubelet, aerpaw17 Readiness probe failed: HTTP probe failed with statuscode: 503
Warning FailedCreatePodSandBox 7m31s kubelet, aerpaw17 Failed create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "85032397a0f2c6b6c86aa9510cdf7ae27e90c415d9908932b7489306dd7576a6" network for pod "coredns-5644d7b6d9-twr2q": networkPlugin cni failed to set up pod "coredns-5644d7b6d9-twr2q_kube-system" network: open /run/flannel/subnet.env: no such file or directory, failed to clean up sandbox container "85032397a0f2c6b6c86aa9510cdf7ae27e90c415d9908932b7489306dd7576a6" network for pod "coredns-5644d7b6d9-twr2q": networkPlugin cni failed to teardown pod "coredns-5644d7b6d9-twr2q_kube-system" network: fork/exec /opt/cni/bin/flannel: text file busy]
Warning FailedCreatePodSandBox 7m30s kubelet, aerpaw17 Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "0961e5369fdb041d6ccc84c5c70de16bbce37143798f275716c4fe11c457f850" network for pod "coredns-5644d7b6d9-twr2q": networkPlugin cni failed to set up pod "coredns-5644d7b6d9-twr2q_kube-system" network: open /run/flannel/subnet.env: no such file or directory
Warning FailedCreatePodSandBox 7m29s kubelet, aerpaw17 Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "5032a83ded37f0dfbfa4e93222c6f62796f5af885ef2f722fb1d1c22b25db79f" network for pod "coredns-5644d7b6d9-twr2q": networkPlugin cni failed to set up pod "coredns-5644d7b6d9-twr2q_kube-system" network: open /run/flannel/subnet.env: no such file or directory
Warning FailedCreatePodSandBox 7m28s kubelet, aerpaw17 Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "00106dc0a7a73b0698838534baf2d5f9d44549f73eb47cbb1d4e14d8e70d11be" network for pod "coredns-5644d7b6d9-twr2q": networkPlugin cni failed to set up pod "coredns-5644d7b6d9-twr2q_kube-system" network: open /run/flannel/subnet.env: no such file or directory
Normal SandboxChanged 7m27s (x5 over 7m32s) kubelet, aerpaw17 Pod sandbox changed, it will be killed and re-created.
Normal Pulled 7m27s kubelet, aerpaw17 Container image "k8s.gcr.io/coredns:1.6.2" already present on machine
Normal Created 7m27s kubelet, aerpaw17 Created container coredns
Normal Started 7m27s kubelet, aerpaw17 Started container coredns
Warning Unhealthy 2m20s (x31 over 7m20s) kubelet, aerpaw17 Readiness probe failed: HTTP probe failed with statuscode: 503

[joshuamoorexyz]

Are you using virtualbox or similar instead of bare metal machine?

[shrsam]

I am using an Intel NUC device running on Ubuntu

[sareek]

I am using an Intel NUC device running on Ubuntu

Try restarting the pod with "sudo kubectl rollout restart deployment -n kube-system"

replace with coredns

Let me know if you have already resolved this issue in another way.

[joshuamoorexyz]

@shrsam Has this been resolved?