Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kube-dns-3841192733-98vm5 0/3 CrashLoopBackOff #1494

Closed
pennpeng opened this issue Aug 3, 2017 · 3 comments
Closed

kube-dns-3841192733-98vm5 0/3 CrashLoopBackOff #1494

pennpeng opened this issue Aug 3, 2017 · 3 comments

Comments

@pennpeng
Copy link

pennpeng commented Aug 3, 2017 •
edited
Loading

Environment:

  • vmware workstation 12

  • OS (Linux 3.10.0-514.26.2.el7.x86_64 x86_64\n"):

[root@k8s-master kubernetes]# cat /etc/os-release 
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
  • Version of Ansible (ansible --version):
[root@k8s-master kubernetes]# ansible --version
ansible 2.3.1.0
  config file = 
  configured module search path = Default w/o overrides
  python version = 2.7.5 (default, Nov  6 2016, 00:28:07) [GCC 4.8.5 20150623 (Red Hat 4.8.5-11)]

Kubespray version (commit) (git rev-parse --short HEAD):
use ansible-playbook to deploy.

Network plugin used:
default calico

Copy of your inventory file:

[kube-master]
# node1
# node2
k8s-master

[etcd]
# node1
# node2
# node3
k8s-master

[kube-node]
# node2
# node3
# node4
# node5
# node6
k8s-node1
k8s-node2

[k8s-cluster:children]
kube-node
kube-master

Command used to invoke ansible:

ansible-playbook -i inventory/inventory cluster.yml  --skip-tags=download

Output of ansible run:


PLAY RECAP *******************************************************************************************************************************************************************************************************************************************************************
k8s-master                 : ok=193  changed=40   unreachable=0    failed=0   
k8s-node1                  : ok=173  changed=44   unreachable=0    failed=0   
k8s-node2                  : ok=141  changed=32   unreachable=0    failed=0   
localhost                  : ok=3    changed=0    unreachable=0    failed=0

The question：

[root@k8s-master kubespray]# kubectl get nodes
NAME         STATUS                     AGE       VERSION
k8s-master   Ready,SchedulingDisabled   1h        v1.6.7+coreos.0
k8s-node1    Ready                      1h        v1.6.7+coreos.0
k8s-node2    Ready                      1h        v1.6.7+coreos.0

[root@k8s-master kubespray]# kubectl get pod --namespace=kube-system 
NAME                                  READY     STATUS             RESTARTS   AGE
kube-apiserver-k8s-master             1/1       Running            0          1h
kube-controller-manager-k8s-master    1/1       Running            0          1h
kube-dns-3841192733-3gt4j             3/3       Running            0          1h
kube-dns-3841192733-98vm5             1/3       CrashLoopBackOff   74         1h
kube-proxy-k8s-master                 1/1       Running            0          1h
kube-proxy-k8s-node1                  1/1       Running            0          1h
kube-proxy-k8s-node2                  1/1       Running            1          1h
kube-scheduler-k8s-master             1/1       Running            0          1h
kubedns-autoscaler-1833630871-kl6h4   1/1       Running            0          1h
nginx-proxy-k8s-node1                 1/1       Running            0          1h
nginx-proxy-k8s-node2                 1/1       Running            1          1h

[root@k8s-master kubespray]# kubectl describe pod kube-dns-3841192733-98vm5  --namespace=kube-system 
Name:		kube-dns-3841192733-98vm5
Namespace:	kube-system
Node:		k8s-node2/192.168.234.152
Start Time:	Thu, 03 Aug 2017 10:55:13 +0800
Labels:		k8s-app=kube-dns
		pod-template-hash=3841192733
Annotations:	kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"kube-system","name":"kube-dns-3841192733","uid":"26f7bd9e-77f7-11e7-b315-000c2973...
		scheduler.alpha.kubernetes.io/critical-pod=
Status:		Running
IP:		10.233.67.1
Controllers:	ReplicaSet/kube-dns-3841192733
Containers:
  kubedns:
    Container ID:	docker://a81d06f91f4f9258a206ca6c52b06a1580da0549f9349e2e7f39b7eff9ced52f
    Image:		gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.2
    Image ID:		docker-pullable://registry.cn-hangzhou.aliyuncs.com/golden/k8s-dns-kube-dns-amd64@sha256:a19dc86abee24d5b612419d5eb1f6590e0e43fd06eed45d6c01551e894bc98e8
    Ports:		10053/UDP, 10053/TCP, 10055/TCP
    Args:
      --domain=cluster.local.
      --dns-port=10053
      --config-dir=/kube-dns-config
      --v=2
    State:		Waiting
      Reason:		CrashLoopBackOff
    Last State:		Terminated
      Reason:		Error
      Exit Code:	255
      Started:		Thu, 03 Aug 2017 12:18:07 +0800
      Finished:		Thu, 03 Aug 2017 12:19:07 +0800
    Ready:		False
    Restart Count:	22
    Limits:
      memory:	170Mi
    Requests:
      cpu:	100m
      memory:	70Mi
    Liveness:	http-get http://:10054/healthcheck/kubedns delay=60s timeout=5s period=10s #success=1 #failure=5
    Readiness:	http-get http://:8081/readiness delay=3s timeout=5s period=10s #success=1 #failure=3
    Environment:
      PROMETHEUS_PORT:	10055
    Mounts:
      /kube-dns-config from kube-dns-config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-3zdmd (ro)
  dnsmasq:
    Container ID:	docker://9f4a206ba43c47574ebcf740c13529737c5750733e7bda19ae6b4261876c0e78
    Image:		gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.2
    Image ID:		docker-pullable://registry.cn-hangzhou.aliyuncs.com/google-containers/k8s-dns-dnsmasq-nanny-amd64@sha256:70b157a6695e5dddc5a3741c401c0507f0fd9326376c77162c690f5f5e82a264
    Ports:		53/UDP, 53/TCP
    Args:
      -v=2
      -logtostderr
      -configDir=/etc/k8s/dns/dnsmasq-nanny
      -restartDnsmasq=true
      --
      -k
      --cache-size=1000
      --log-facility=-
      --server=/cluster.local/127.0.0.1#10053
      --server=/in-addr.arpa/127.0.0.1#10053
      --server=/ip6.arpa/127.0.0.1#10053
    State:		Running
      Started:		Thu, 03 Aug 2017 12:19:41 +0800
    Last State:		Terminated
      Reason:		Error
      Exit Code:	137
      Started:		Thu, 03 Aug 2017 12:12:47 +0800
      Finished:		Thu, 03 Aug 2017 12:14:27 +0800
    Ready:		True
    Restart Count:	25
    Requests:
      cpu:		150m
      memory:		20Mi
    Liveness:		http-get http://:10054/healthcheck/dnsmasq delay=60s timeout=5s period=10s #success=1 #failure=5
    Environment:	<none>
    Mounts:
      /etc/k8s/dns/dnsmasq-nanny from kube-dns-config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-3zdmd (ro)
  sidecar:
    Container ID:	docker://790b80b09ed34a792802f53e9f2b230f56073e647cac6052b23e32493f21f803
    Image:		gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.2
    Image ID:		docker-pullable://registry.cn-hangzhou.aliyuncs.com/google-containers/k8s-dns-sidecar-amd64@sha256:7568f2adae47ebac0e40bc3fd6358b27e5c8894627d2d48e7748e159b02403ac
    Port:		10054/TCP
    Args:
      --v=2
      --logtostderr
      --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,A
      --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,A
    State:		Waiting
      Reason:		CrashLoopBackOff
    Last State:		Terminated
      Reason:		Error
      Exit Code:	2
      Started:		Thu, 03 Aug 2017 12:14:37 +0800
      Finished:		Thu, 03 Aug 2017 12:15:47 +0800
    Ready:		False
    Restart Count:	27
    Requests:
      cpu:		10m
      memory:		20Mi
    Liveness:		http-get http://:10054/metrics delay=60s timeout=5s period=10s #success=1 #failure=5
    Environment:	<none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-3zdmd (ro)
Conditions:
  Type		Status
  Initialized 	True 
  Ready 	False 
  PodScheduled 	True 
Volumes:
  kube-dns-config:
    Type:	ConfigMap (a volume populated by a ConfigMap)
    Name:	kube-dns
    Optional:	true
  default-token-3zdmd:
    Type:	Secret (a volume populated by a Secret)
    SecretName:	default-token-3zdmd
    Optional:	false
QoS Class:	Burstable
Node-Selectors:	<none>
Tolerations:	CriticalAddonsOnly=:Exists
Events:
  FirstSeen	LastSeen	Count	From			SubObjectPath	Type		Reason		Message
  ---------	--------	-----	----			-------------	--------	------		-------
  59m		59m		2	kubelet, k8s-node2			Warning		FailedSync	Error syncing pod, skipping: failed to "StartContainer" for "kubedns" with CrashLoopBackOff: "Back-off 20s restarting failed container=kubedns pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"

  59m	59m	1	kubelet, k8s-node2	spec.containers{sidecar}	Normal	Killing		Killing container with docker id 3c9b0b052514: pod "kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)" container "sidecar" is unhealthy, it will be killed and re-created.
  59m	59m	1	kubelet, k8s-node2	spec.containers{kubedns}	Normal	Started		Started container with docker id 5db552060a2b
  59m	59m	1	kubelet, k8s-node2	spec.containers{kubedns}	Normal	Created		Created container with docker id 5db552060a2b; Security:[seccomp=unconfined]
  59m	59m	1	kubelet, k8s-node2	spec.containers{dnsmasq}	Normal	Killing		Killing container with docker id 7bef7de03d6d: pod "kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)" container "dnsmasq" is unhealthy, it will be killed and re-created.
  59m	59m	1	kubelet, k8s-node2	spec.containers{dnsmasq}	Normal	Started		Started container with docker id 323dd27a01d2
  59m	59m	1	kubelet, k8s-node2	spec.containers{dnsmasq}	Normal	Created		Created container with docker id 323dd27a01d2; Security:[seccomp=unconfined]
  58m	58m	1	kubelet, k8s-node2	spec.containers{sidecar}	Normal	Killing		Killing container with docker id dd9c6256b2e7: pod "kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)" container "sidecar" is unhealthy, it will be killed and re-created.
  57m	57m	1	kubelet, k8s-node2	spec.containers{dnsmasq}	Normal	Killing		Killing container with docker id 323dd27a01d2: pod "kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)" container "dnsmasq" is unhealthy, it will be killed and re-created.
  58m	57m	3	kubelet, k8s-node2					Warning	FailedSync	Error syncing pod, skipping: failed to "StartContainer" for "kubedns" with CrashLoopBackOff: "Back-off 40s restarting failed container=kubedns pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"

  56m	56m	1	kubelet, k8s-node2	spec.containers{sidecar}	Normal	Killing		Killing container with docker id 5987337b15c1: pod "kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)" container "sidecar" is unhealthy, it will be killed and re-created.
  55m	55m	1	kubelet, k8s-node2	spec.containers{dnsmasq}	Normal	Killing		Killing container with docker id 7060918cd885: pod "kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)" container "dnsmasq" is unhealthy, it will be killed and re-created.
  56m	55m	7	kubelet, k8s-node2					Warning	FailedSync	Error syncing pod, skipping: failed to "StartContainer" for "kubedns" with CrashLoopBackOff: "Back-off 1m20s restarting failed container=kubedns pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"

  54m	54m	1	kubelet, k8s-node2	spec.containers{sidecar}	Normal	Killing		Killing container with docker id e001cc7ecc3d: pod "kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)" container "sidecar" is unhealthy, it will be killed and re-created.
  54m	53m	5	kubelet, k8s-node2					Warning	FailedSync	Error syncing pod, skipping: failed to "StartContainer" for "kubedns" with CrashLoopBackOff: "Back-off 2m40s restarting failed container=kubedns pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"

  52m	52m	3	kubelet, k8s-node2		Warning	FailedSync	Error syncing pod, skipping: [failed to "StartContainer" for "dnsmasq" with CrashLoopBackOff: "Back-off 2m40s restarting failed container=dnsmasq pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"
, failed to "StartContainer" for "sidecar" with CrashLoopBackOff: "Back-off 2m40s restarting failed container=sidecar pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"
, failed to "StartContainer" for "kubedns" with CrashLoopBackOff: "Back-off 2m40s restarting failed container=kubedns pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"
]
  52m	51m	4	kubelet, k8s-node2		Warning	FailedSync	Error syncing pod, skipping: [failed to "StartContainer" for "kubedns" with CrashLoopBackOff: "Back-off 2m40s restarting failed container=kubedns pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"
, failed to "StartContainer" for "dnsmasq" with CrashLoopBackOff: "Back-off 2m40s restarting failed container=dnsmasq pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"
, failed to "StartContainer" for "sidecar" with CrashLoopBackOff: "Back-off 2m40s restarting failed container=sidecar pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"
]
  51m	50m	6	kubelet, k8s-node2		Warning	FailedSync	Error syncing pod, skipping: [failed to "StartContainer" for "dnsmasq" with CrashLoopBackOff: "Back-off 2m40s restarting failed container=dnsmasq pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"
, failed to "StartContainer" for "sidecar" with CrashLoopBackOff: "Back-off 2m40s restarting failed container=sidecar pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"
]
  50m	50m	2	kubelet, k8s-node2		Warning	FailedSync	Error syncing pod, skipping: [failed to "StartContainer" for "kubedns" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=kubedns pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"
, failed to "StartContainer" for "dnsmasq" with CrashLoopBackOff: "Back-off 2m40s restarting failed container=dnsmasq pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"
, failed to "StartContainer" for "sidecar" with CrashLoopBackOff: "Back-off 2m40s restarting failed container=sidecar pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"
]
  50m	50m	3	kubelet, k8s-node2		Warning	FailedSync	Error syncing pod, skipping: [failed to "StartContainer" for "kubedns" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=kubedns pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"
, failed to "StartContainer" for "dnsmasq" with CrashLoopBackOff: "Back-off 2m40s restarting failed container=dnsmasq pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"
]
  44m	44m	1	kubelet, k8s-node2		Warning	FailedSync	Error syncing pod, skipping: [failed to "StartContainer" for "sidecar" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=sidecar pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"
, failed to "StartContainer" for "dnsmasq" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=dnsmasq pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"
]
  1h	44m	3	kubelet, k8s-node2	spec.containers{kubedns}	Warning	Unhealthy	Liveness probe failed: Get http://10.233.67.1:10054/healthcheck/kubedns: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  20m	13m	12	kubelet, k8s-node2					Warning	FailedSync	Error syncing pod, skipping: failed to "StartContainer" for "dnsmasq" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=dnsmasq pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"

  34m	8m	4	kubelet, k8s-node2		Warning	FailedSync	Error syncing pod, skipping: [failed to "StartContainer" for "sidecar" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=sidecar pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"
, failed to "StartContainer" for "kubedns" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=kubedns pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"
]
  32m	7m	15	kubelet, k8s-node2		Warning	FailedSync	Error syncing pod, skipping: failed to "StartContainer" for "sidecar" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=sidecar pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"

  49m	6m	16	kubelet, k8s-node2		Warning	FailedSync	Error syncing pod, skipping: failed to "StartContainer" for "kubedns" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=kubedns pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"

  1h	6m	21	kubelet, k8s-node2	spec.containers{dnsmasq}	Warning	Unhealthy	Liveness probe failed: Get http://10.233.67.1:10054/healthcheck/dnsmasq: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  1h	5m	19	kubelet, k8s-node2	spec.containers{sidecar}	Normal	Pulled		Container image "gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.2" already present on machine
  12m	5m	5	kubelet, k8s-node2					Warning	FailedSync	Error syncing pod, skipping: [failed to "StartContainer" for "dnsmasq" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=dnsmasq pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"
, failed to "StartContainer" for "kubedns" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=kubedns pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"
]
  44m	4m	51	kubelet, k8s-node2		Warning	FailedSync	Error syncing pod, skipping: [failed to "StartContainer" for "kubedns" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=kubedns pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"
, failed to "StartContainer" for "dnsmasq" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=dnsmasq pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"
]
  54m	4m	27	kubelet, k8s-node2	spec.containers{dnsmasq}	Normal	Killing		(events with common reason combined)
  1h	4m	23	kubelet, k8s-node2	spec.containers{sidecar}	Warning	Unhealthy	Liveness probe failed: Get http://10.233.67.1:10054/metrics: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  49m	4m	31	kubelet, k8s-node2					Warning	FailedSync	(events with common reason combined)
  1h	2m	15	kubelet, k8s-node2	spec.containers{kubedns}	Normal	Pulled		Container image "gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.2" already present on machine
  45m	1m	17	kubelet, k8s-node2					Warning	FailedSync	Error syncing pod, skipping: [failed to "StartContainer" for "dnsmasq" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=dnsmasq pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"
, failed to "StartContainer" for "sidecar" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=sidecar pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"
]
  1h	1m	91	kubelet, k8s-node2	spec.containers{kubedns}	Warning	Unhealthy	Readiness probe failed: Get http://10.233.67.1:8081/readiness: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  46m	1m	6	kubelet, k8s-node2					Warning	FailedSync	Error syncing pod, skipping: [failed to "StartContainer" for "dnsmasq" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=dnsmasq pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"
, failed to "StartContainer" for "sidecar" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=sidecar pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"
, failed to "StartContainer" for "kubedns" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=kubedns pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"
]
  45m	1m	11	kubelet, k8s-node2		Warning	FailedSync	Error syncing pod, skipping: [failed to "StartContainer" for "sidecar" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=sidecar pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"
, failed to "StartContainer" for "kubedns" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=kubedns pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"
, failed to "StartContainer" for "dnsmasq" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=dnsmasq pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"
]
  48m	1m	57	kubelet, k8s-node2		Warning	FailedSync	Error syncing pod, skipping: [failed to "StartContainer" for "kubedns" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=kubedns pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"
, failed to "StartContainer" for "dnsmasq" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=dnsmasq pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"
, failed to "StartContainer" for "sidecar" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=sidecar pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"
]
  59m	45s	43	kubelet, k8s-node2	spec.containers{sidecar}	Normal	Started		(events with common reason combined)
  59m	45s	43	kubelet, k8s-node2	spec.containers{sidecar}	Normal	Created		(events with common reason combined)
  1h	45s	18	kubelet, k8s-node2	spec.containers{dnsmasq}	Normal	Pulled		Container image "gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.2" already present on machine
  49m	16s	49	kubelet, k8s-node2					Warning	FailedSync	Error syncing pod, skipping: [failed to "StartContainer" for "kubedns" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=kubedns pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"
, failed to "StartContainer" for "sidecar" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=sidecar pod=kube-dns-3841192733-98vm5_kube-system(2c5a8946-77f7-11e7-b315-000c29735600)"
]
  59m	16s	647	kubelet, k8s-node2	spec.containers{kubedns}	Warning	BackOff	Back-off restarting failed docker container
[root@k8s-master kubespray]#

the logs:

[root@k8s-master kubespray]# kubectl logs kube-dns-3841192733-98vm5 -c kubedns  --namespace=kube-system 
I0803 04:18:07.891782       1 dns.go:48] version: 1.14.1-16-gff416ee
I0803 04:18:07.895104       1 server.go:70] Using configuration read from directory: /kube-dns-config with period 10s
I0803 04:18:07.895201       1 server.go:113] FLAG: --alsologtostderr="false"
I0803 04:18:07.895231       1 server.go:113] FLAG: --config-dir="/kube-dns-config"
I0803 04:18:07.895239       1 server.go:113] FLAG: --config-map=""
I0803 04:18:07.895243       1 server.go:113] FLAG: --config-map-namespace="kube-system"
I0803 04:18:07.895246       1 server.go:113] FLAG: --config-period="10s"
I0803 04:18:07.895255       1 server.go:113] FLAG: --dns-bind-address="0.0.0.0"
I0803 04:18:07.895259       1 server.go:113] FLAG: --dns-port="10053"
I0803 04:18:07.895265       1 server.go:113] FLAG: --domain="cluster.local."
I0803 04:18:07.895271       1 server.go:113] FLAG: --federations=""
I0803 04:18:07.895278       1 server.go:113] FLAG: --healthz-port="8081"
I0803 04:18:07.895282       1 server.go:113] FLAG: --initial-sync-timeout="1m0s"
I0803 04:18:07.895285       1 server.go:113] FLAG: --kube-master-url=""
I0803 04:18:07.895290       1 server.go:113] FLAG: --kubecfg-file=""
I0803 04:18:07.895295       1 server.go:113] FLAG: --log-backtrace-at=":0"
I0803 04:18:07.895300       1 server.go:113] FLAG: --log-dir=""
I0803 04:18:07.895305       1 server.go:113] FLAG: --log-flush-frequency="5s"
I0803 04:18:07.895308       1 server.go:113] FLAG: --logtostderr="true"
I0803 04:18:07.895315       1 server.go:113] FLAG: --nameservers=""
I0803 04:18:07.895318       1 server.go:113] FLAG: --stderrthreshold="2"
I0803 04:18:07.895321       1 server.go:113] FLAG: --v="2"
I0803 04:18:07.895325       1 server.go:113] FLAG: --version="false"
I0803 04:18:07.895332       1 server.go:113] FLAG: --vmodule=""
I0803 04:18:07.895458       1 server.go:176] Starting SkyDNS server (0.0.0.0:10053)
I0803 04:18:07.895810       1 server.go:198] Skydns metrics enabled (/metrics:10055)
I0803 04:18:07.895827       1 dns.go:147] Starting endpointsController
I0803 04:18:07.895831       1 dns.go:150] Starting serviceController
I0803 04:18:07.895908       1 logs.go:41] skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053 [rcache 0]
I0803 04:18:07.895917       1 logs.go:41] skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053 [rcache 0]
I0803 04:18:08.397222       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:08.896880       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:09.396930       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:09.897192       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:10.396988       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:10.896985       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
E0803 04:18:10.902413       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: no route to host
E0803 04:18:10.902554       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: no route to host
I0803 04:18:11.397083       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:11.897059       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:12.396887       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:12.896929       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:13.397139       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:13.897123       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
E0803 04:18:13.908867       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: no route to host
E0803 04:18:13.909156       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: no route to host
I0803 04:18:14.396965       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:14.897073       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:15.397285       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:15.897207       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:16.396174       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:16.897106       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
E0803 04:18:16.914789       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: no route to host
E0803 04:18:16.914923       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: no route to host
I0803 04:18:17.397103       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:17.896291       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:18.396513       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:18.896368       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:19.396581       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:19.897031       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
E0803 04:18:19.921233       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: no route to host
E0803 04:18:19.921256       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: no route to host
I0803 04:18:20.396446       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:20.897215       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:21.396261       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:21.897082       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:22.396869       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:22.896789       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
E0803 04:18:22.931870       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: no route to host
E0803 04:18:22.931869       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: no route to host
I0803 04:18:23.396893       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:23.896658       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:24.396587       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:24.896141       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:25.396507       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:25.896707       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
E0803 04:18:25.936727       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: no route to host
E0803 04:18:25.937316       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: no route to host
I0803 04:18:26.396957       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:26.896977       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:27.397178       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:27.896168       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:28.397031       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:28.897111       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
E0803 04:18:28.942230       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: no route to host
E0803 04:18:28.942279       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: no route to host
I0803 04:18:29.396310       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:29.896850       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:30.397048       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:30.896278       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:31.396456       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:31.896951       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
E0803 04:18:31.949008       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: no route to host
E0803 04:18:31.949023       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: no route to host
I0803 04:18:32.396747       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:32.897225       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:33.396297       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:33.897102       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:34.396412       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:34.896683       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
E0803 04:18:34.955200       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: no route to host
E0803 04:18:34.955272       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: no route to host
I0803 04:18:35.397422       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:35.897046       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:36.396380       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:36.896352       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:37.396763       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:37.896202       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
E0803 04:18:37.961117       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: no route to host
E0803 04:18:37.961153       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: no route to host
I0803 04:18:38.396716       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:38.897047       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:39.397139       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:39.897410       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:40.396337       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:40.896216       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
E0803 04:18:40.967223       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: no route to host
E0803 04:18:40.967261       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: no route to host
I0803 04:18:41.397360       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:41.896264       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:42.396843       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:42.897113       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:43.397013       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:43.896346       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
E0803 04:18:43.973440       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: no route to host
E0803 04:18:43.973518       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: no route to host
I0803 04:18:44.397088       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:44.897010       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:45.396505       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:45.897058       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:46.397338       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:46.897066       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
E0803 04:18:46.980430       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: no route to host
E0803 04:18:46.980462       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: no route to host
I0803 04:18:47.396302       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:47.897045       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:48.397023       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:48.896900       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:49.396560       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:49.897958       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
E0803 04:18:49.988552       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: no route to host
E0803 04:18:49.988583       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: no route to host
I0803 04:18:50.396664       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:50.897112       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:51.397026       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:51.896701       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:52.396782       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:52.896505       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
E0803 04:18:52.997285       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: no route to host
E0803 04:18:52.997301       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: no route to host
I0803 04:18:53.396601       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:53.897182       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:54.397355       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:54.896748       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:55.396319       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:55.896660       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
E0803 04:18:56.004506       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: no route to host
E0803 04:18:56.004644       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: no route to host
I0803 04:18:56.396640       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:56.896921       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:57.396304       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:57.896354       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:58.396517       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:58.896940       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
E0803 04:18:59.010882       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: no route to host
E0803 04:18:59.011046       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: no route to host
I0803 04:18:59.397542       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:18:59.896846       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:19:00.397089       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:19:00.897015       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:19:01.396944       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:19:01.897039       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
E0803 04:19:02.017239       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: no route to host
E0803 04:19:02.017256       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: no route to host
I0803 04:19:02.397158       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:19:02.896773       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:19:03.396850       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:19:03.896908       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:19:04.397151       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:19:04.897193       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
E0803 04:19:05.024809       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: no route to host
E0803 04:19:05.024814       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.233.0.1:443: getsockopt: no route to host
I0803 04:19:05.397319       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:19:05.897037       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:19:06.396963       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:19:06.897126       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0803 04:19:07.396730       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
F0803 04:19:07.896742       1 dns.go:168] Timeout waiting for initialization
@pennpeng pennpeng closed this as completed Aug 3, 2017
@mkjpryor-stfc
Copy link

@pennpeng Did you fix this issue? If so, how? I'm having the same problem.

@SStar1314
Copy link

SStar1314 commented Oct 10, 2017 •
edited
Loading

I hit same issue. Maybe caused by iptables rules. The "10.96.0.1" ip is kubernetes cluster ip address.
The related iptables rules is below:
-A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.96.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.96.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y

On kubernetes node, run "curl https://10.96.0.1:443/api/v1/services?resourceVersion=0 -k" can print something.

--> kubernetes/kubernetes#36835
--> kubernetes/kubeadm#102

@RahulG115
Copy link

iptables flush work for me
just run this command $iptables flush
and delete that dns pod
thanks @SStar1314

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@pennpeng @SStar1314 @mkjpryor-stfc @RahulG115