kubeadm: preflight check false positives in CentOS 7

[mikedanese]

From @jberkus on November 6, 2016 4:30

Platform: CentOS 7.2.1511 on AWS EC2 (t2.medium)

kubeadm version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-alpha.2.380+85fe0f1aadf91e", GitCommit:"85fe0f1aadf91e134102cf3c01a9eed11a7e257f", GitTreeState:"clean", BuildDate:"2016-11-02T14:58:17Z", GoVersion:"go1.7.1", Compiler:"gc", Platform:"linux/amd64"}

Symptoms:

While following installation instructions at http://kubernetes.io/docs/getting-started-guides/kubeadm/, I am finding that I need to always disable preflight checks due to spurious failures.

With kubeadmin --init:

preflight check errors:
	ebtables not found in system path

This error appears on all CentOS machines on AWS I've tried. It doesn't seem to relate to anything at all that I can find; the only google hit I can get on it is a Kubernetes issue explaining how to use --skip-preflight-checks. It does not seem to block kubeadmin setup.

On the nodes I get an additional error:

preflight check errors:
	/etc/kubernetes is not empty
	ebtables not found in system path

Indeed, /etc/kubernetes is not empty ... because the RPMs distributed by Kubernetes.io include a sub-directory in that dir, "manifests".

Copied from original issue: kubernetes/kubernetes#36301

[mikedanese]

From @jberkus on November 6, 2016 21:3

The ebtables false positive does not happen with CentOS 7 Atomic on bare metal. Unfortunately, I don't have a regular Centos 7 on bare metal to test on for comparison.

[mikedanese]

From @luxas on November 6, 2016 23:19

Why just don't install ebtables in case you don't have it?

Basically kubeadm does just which ebtables and returns error if it doesn't exist, since kubelet requires ebtables to be present.

[mikedanese]

From @jberkus on November 7, 2016 0:21

Except that apparently it doesn't, because Kubelet apparently works fine with no ebtables.

Also, if ebtables is required, why isn't it marked as a dependency of kubeadm? Or at least mentioned in the docs?

[mikedanese]

From @luxas on November 7, 2016 1:22

Yes, ebtables is a dependency of the kubelet, see: kubernetes/kubernetes#28717 and kubernetes/kubernetes#26093.

You're right that ebtables doesn't seem to be in the rpm dependencies of the kubelet, sorry for that.
I opened kubernetes/release#197 to minimize the delta between debs and rpms, I added the deps to the debs, but it wasn't reflected in the rpms.
cc: @dgoodwin on the rpms

[mikedanese]

From @wallrj on November 10, 2016 16:14

FYI I'm seeing the pre-flight check error too:

[[email protected].]: Running kubeadm join --token e87b93.ec1e2d95c2bb65ee 104.196.x.x [[email protected].]: Running pre-flight checks
[[email protected].]: WARNING: docker service is not enabled, please run 'systemctl enable docker.service'
[[email protected].]: preflight check errors:
[[email protected].]: 	/etc/kubernetes is not empty
cluster provisioning failed

 Dependencies Resolved
 
 ================================================================================
  Package     Arch   Version                              Repository        Size
 ================================================================================
 Installing:
  kubeadm     x86_64 1.5.0-1.alpha.2.380.85fe0f1aadf91e.0 kubernetes       9.7 M
  kubectl     x86_64 1.4.4-1                              google-cloud-sdk 9.1 M
  kubelet     x86_64 1.4.4-1                              kubernetes        13 M
  kubernetes-cni
              x86_64 0.3.0.1-1.07a8a2                     kubernetes       9.8 M
 Installing for dependencies:
  socat       x86_64 1.7.2.2-5.el7                        base             255 k
 
 Transaction Summary
 ================================================================================
 Install  4 Packages (+1 Dependent package)

[mikedanese]

From @dgoodwin on November 10, 2016 18:25

@wallrj this was fixed in fe03e1319f6d279f6b9ac60c8ca34fe907eb94e3 I believe, but it did not make the build your output.

[mikedanese]

From @luxas on November 11, 2016 21:45

@mikedanese Please push the rpms you built from kubernetes/release#197, so this will be fixed
I think you started building them on Monday already...

[mikedanese]

From @waynebrantley on November 15, 2016 13:43

etables is easy - just install the package.
However, what about:

preflight check errors:
	/etc/kubernetes is not empty

[mikedanese]

From @dgoodwin on November 15, 2016 13:52

@waynebrantley fixed but not released yet, details above.

[mikedanese]

From @waynebrantley on November 15, 2016 13:55

Ok, I just ran it with --skip-preflight-checks
FYI, I did a kubeadm reset on a new node....
Then did a kubeadm join......the node never joined. I rolled back to before that and join worked fine.
This may be a related issue?

[mikedanese]

From @dgoodwin on November 15, 2016 14:1

Probably very unlikely to be related, this preflight check is just a simple bad assumption that /etc/kubernetes doesn't exist or is completely empty, but the packages create /etc/kubernetes/manifests and the check always fails by default.

[mikedanese]

From @jberkus on November 15, 2016 20:13

Unlikely to be related.

[mikedanese]

From @mariano-dim on November 21, 2016 17:30

you have to delete the content of /etc/kubernetes and try again

[jberkus]

Just tested the latest packages. False positives appear to be fixed. As such, this issue can be closed. However, I can't close it because I don't own it anymore.

[luxas]

Ok, thanks @jberkus!