kubeadm failed, kubelet:failed pulling images: Client.Timeout exceeded while awaiting headers

[ayeteng]

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

Please provide the following details:

Environment:

Minikube version (use minikube version):

• OS (e.g. from /etc/os-release): mac OS High Sierra 10.13.5
• VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName): hyperkit
• ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep -i ISO or minikube ssh cat /etc/VERSION): minikube-v0.28.0.iso
• Install tools:
• Others:
  The above can be generated in one go with the following commands (can be copied and pasted directly into your terminal):

minikube version
echo "";
echo "OS:";
cat /etc/os-release
echo "";
echo "VM driver": 
grep DriverName ~/.minikube/machines/minikube/config.json
echo "";
echo "ISO version";
grep -i ISO ~/.minikube/machines/minikube/config.json

What happened:

minikube start --vm-driver=hyperkit
Starting local Kubernetes v1.10.0 cluster...
Starting VM...
Downloading Minikube ISO
153.08 MB / 153.08 MB [============================================] 100.00% 0s
Getting VM IP address...
Moving files into cluster...
Downloading kubeadm v1.10.0
Downloading kubelet v1.10.0
Finished Downloading kubelet v1.10.0
Finished Downloading kubeadm v1.10.0
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
E0703 12:53:57.980231 9743 start.go:294] Error starting cluster: kubeadm init error sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data-minikube --ignore-preflight-errors=Port-10250 --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI running command: : running command: sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data-minikube --ignore-preflight-errors=Port-10250 --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI
.: Process exited with status 1
================================================================================
An error has occurred. Would you like to opt in to sending anonymized crash
information to minikube to help prevent future errors?
To opt out of these messages, run the command:
minikube config set WantReportErrorPrompt false
================================================================================
Please enter your response [Y/n]:

What you expected to happen:
Expect the cluster up and running.
How to reproduce it (as minimally and precisely as possible):
CMD: minikube start --vm-driver=hyperkit
Output of minikube logs (if applicable):
error.txt

Anything else do we need to know:

[alexfrancavilla]

My team is having the exact same issue (getsockopt spam in logs). Everyone is running High Sierra, minikube version v0.26.1, hyperkit v0.20171204-60-g0e5b6b. It magically works just on my machine. Feels like a networking issue that the VM can't connect to the internet properly. If anyone got an idea on what do to, I'd be super grateful.

[irfn]

i faced this as well. The start times out with this error.

E0704 12:24:07.032372   15129 start.go:299] Error restarting cluster:  restarting kube-proxy: waiting for kube-proxy to be up for configmap update: timed out waiting for the condition

On restart (stop,start) i am seeing same error.
i am on High Sierra [10.13.5 (17F77)] Virtualbox 5.2.6

[irfn]

Deleting the vm and starting over worked.
minikube delete
minikube start
This is possibly some api version compatibility issue

[alexfrancavilla]

@irfn I believe that your issue is something different. On my team the established workaround is to switch to VirtualBox when hyperkit gets stuck. Every now and then the apiserver then becomes unreachable and pods need to be restarted, never faced these issues with hyperkit though.

[ayeteng]

@irfn The command you pointed out was not resolved the issue.
my minikube works fine with virtualbox if I delete the cluster and start the minikube.
my minikube doesn't work with hyperkit even I delete the cluster and then start it.
It was worded with hyperkit in very first deployment. But somehow my system crashed. I restart the minikube then I cannot get cluster up and runing with hyperkit anymore. Any idea to delete the crashed cluster data? Tried rm -rf ~/.minikube but it doesn't back to normal.

[KateJC]

hello , minikube cannot start 。How should I solve it ，please give me help.

in the Centos7
[root@localhost bin]# minikube version
minikube version: v0.28.0
[root@localhost bin]# kubectl version
-bash: /usr/local/bin/kubectl: cannot execute binary file
[root@localhost bin]#

[root@localhost bin]# minikube start
Starting local Kubernetes v1.10.0 cluster...
Starting VM...
Downloading Minikube ISO
153.08 MB / 153.08 MB [============================================] 100.00% 0s
Getting VM IP address...
Moving files into cluster...
Downloading kubeadm v1.10.0
Downloading kubelet v1.10.0
E0716 22:17:35.806802 1833 start.go:252] Error updating cluster: downloading binaries: downloading kubeadm: Error downloading kubeadm v1.10.0: failed to download: failed to download to temp file: download failed: 5 error(s) occurred:

• Temporary download error: Get https://storage.googleapis.com/kubernetes-release/release/v1.10.0/bin/linux/amd64/kubeadm: net/http: TLS handshake timeout
• Temporary download error: Get https://storage.googleapis.com/kubernetes-release/release/v1.10.0/bin/linux/amd64/kubeadm: net/http: TLS handshake timeout
• Temporary download error: Get https://storage.googleapis.com/kubernetes-release/release/v1.10.0/bin/linux/amd64/kubeadm: net/http: TLS handshake timeout
• Temporary download error: Get https://storage.googleapis.com/kubernetes-release/release/v1.10.0/bin/linux/amd64/kubeadm: dial tcp 172.217.25.16:443: i/o timeout
• Temporary download error: Get https://storage.googleapis.com/kubernetes-release/release/v1.10.0/bin/linux/amd64/kubeadm: dial tcp 172.217.25.16:443: i/o timeout
  E0716 22:18:01.036212 1833 util.go:151] Error uploading error message: : Post https://clouderrorreporting.googleapis.com/v1beta1/projects/k8s-minikube/events:report?key=AIzaSyACUwzG0dEPcl-eOgpDKnyKoUFgHdfoFuA: read tcp 192.168.43.100:33806->172.217.24.42:443: read: connection reset by peer

[tstromberg]

The important errors from the log - indicate a crashing apiserver. If someone runs into this again, can they attach the output of minikube ssh "journalctl -xe". Thanks!

[tstromberg]

I'm pretty sure this is related in some way to network connectivity issues, but I'm not absolutely certain. If you run into this, please attach the output of:

minikube logs
kubectl get pods --all-namespaces
minikube ssh "journalctl -xe"

Thanks!

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[tstromberg]

Keeping open, as this issue is representative of symptoms when the VM can't connect to the internet. Typically because of a firewall being in the way, and occasionally requiring that a proxy be configured within the VM itself.

Some related bugs: #3145 and #3242

[tstromberg]

Issue is now documented - and appears quite a bit differently in newer releases of minikube:

https://github.com/kubernetes/minikube/blob/master/docs/http_proxy.md#unable-to-pull-imagesclienttimeout-exceeded-while-awaiting-headers