bootkube.service - Worker node does not terminate in bootstrap #704

eduardolucioac · 2021-06-19T00:14:37Z

eduardolucioac
Jun 19, 2021

I'm trying to deploy the OKD cluster, but the worker node doesn't finish its installation in the bootstrap process.

Worker node error output:

[   39.613875] ignition[526]: GET https://api-int.mbr.okd.local:22623/config/worker: attempt #11
[   39.621547] ignition[526]: GET result: Internal Server Error

Bootstrap node error output:

[core@localhost ~]$ journalctl -b -f -u release-image.service -u bootkube.service
-- Journal begins at Fri 2021-06-18 21:06:14 UTC. --
Jun 18 23:00:16 localhost bootkube.sh[36767]: Skipped "etcd-ca-bundle-configmap.yaml" configmaps.v1./etcd-ca-bundle -n openshift-config as it already exists
Jun 18 23:00:16 localhost bootkube.sh[36767]: Skipped "etcd-client-secret.yaml" secrets.v1./etcd-client -n openshift-config as it already exists
Jun 18 23:00:43 localhost bootkube.sh[36767]:         Pod Status:openshift-kube-apiserver/kube-apiserver        DoesNotExist
Jun 18 23:00:43 localhost bootkube.sh[36767]:         Pod Status:openshift-kube-scheduler/openshift-kube-scheduler        Ready
Jun 18 23:00:43 localhost bootkube.sh[36767]:         Pod Status:openshift-kube-controller-manager/kube-controller-manager        Ready
Jun 18 23:00:43 localhost bootkube.sh[36767]:         Pod Status:openshift-cluster-version/cluster-version-operator        Failed
Jun 18 23:00:48 localhost bootkube.sh[36767]:         Pod Status:openshift-kube-apiserver/kube-apiserver        DoesNotExist
Jun 18 23:00:48 localhost bootkube.sh[36767]:         Pod Status:openshift-kube-scheduler/openshift-kube-scheduler        Pending
Jun 18 23:00:48 localhost bootkube.sh[36767]:         Pod Status:openshift-kube-controller-manager/kube-controller-manager        Pending
Jun 18 23:00:48 localhost bootkube.sh[36767]:         Pod Status:openshift-cluster-version/cluster-version-operator        Failed

I would appreciate some help here friends!

Thanks! =D

vrutkovs · 2021-06-19T08:10:07Z

vrutkovs
Jun 19, 2021
Maintainer

Please attach or link to log bundle

1 reply

eduardolucioac Jun 19, 2021
Author

@vrutkovs

As you asked... =D

log-bundle-20210619133803.tar.gz

Thanks!

eduardolucioac · 2021-06-19T18:07:28Z

eduardolucioac
Jun 19, 2021
Author

@vrutkovs and anyone who can help... =D

Apart from the log sent (more above), I think I discovered some other informations...

The service on "api" on port "6443" ( https://localhost:6443/api ) is not starting as we can see in these snippets of logs from the service "bootkube.service"...

[core@localhost ~]$ journalctl -b -f -u release-image.service -u bootkube.service
[...]
Jun 19 17:17:03 localhost bootkube.sh[1314]: Starting cluster-bootstrap...
[...]
Jun 19 17:17:47 localhost bootkube.sh[5122]: [#187] failed to fetch discovery: Get "https://localhost:6443/api?timeout=32s": dial tcp [::1]:6443: connect: connection refused
Jun 19 17:17:47 localhost bootkube.sh[5122]: [#188] failed to fetch discovery: Get "https://localhost:6443/api?timeout=32s": dial tcp [::1]:6443: connect: connection refused
Jun 19 17:17:47 localhost bootkube.sh[5122]: [#189] failed to fetch discovery: Get "https://localhost:6443/api?timeout=32s": dial tcp [::1]:6443: connect: connection refused
Jun 19 17:17:47 localhost bootkube.sh[5122]: [#190] failed to fetch discovery: Get "https://localhost:6443/api?timeout=32s": dial tcp [::1]:6443: connect: connection refused
Jun 19 17:17:47 localhost bootkube.sh[5122]: [#191] failed to fetch discovery: Get "https://localhost:6443/api?timeout=32s": dial tcp [::1]:6443: connect: connection refused
Jun 19 17:17:48 localhost bootkube.sh[5122]: [#192] failed to fetch discovery: Get "https://localhost:6443/api?timeout=32s": dial tcp [::1]:6443: connect: connection refused
[...]
Jun 19 17:19:00 localhost bootkube.sh[5122]: Created "99-okd-worker-disable-mitigations.yaml" machineconfigs.v1.machineconfiguration.openshift.io/99-okd-worker-disable-mitigations -n
Jun 19 17:19:00 localhost bootkube.sh[5122]: Failed to create "99_openshift-cluster-api_master-user-data-secret.yaml" secrets.v1./master-user-data -n openshift-machine-api: namespaces "openshift-machine-api" not found
Jun 19 17:19:00 localhost bootkube.sh[5122]: Failed to create "99_openshift-cluster-api_worker-user-data-secret.yaml" secrets.v1./worker-user-data -n openshift-machine-api: namespaces "openshift-machine-api" not found
Jun 19 17:19:00 localhost bootkube.sh[5122]: Created "99_openshift-machineconfig_99-master-ssh.yaml" machineconfigs.v1.machineconfiguration.openshift.io/99-master-ssh -n
Jun 19 17:19:00 localhost bootkube.sh[5122]: Created "99_openshift-machineconfig_99-worker-ssh.yaml" machineconfigs.v1.machineconfiguration.openshift.io/99-worker-ssh -n
Jun 19 17:19:00 localhost bootkube.sh[5122]: [#200] failed to create some manifests:
Jun 19 17:19:00 localhost bootkube.sh[5122]: "99_openshift-cluster-api_master-user-data-secret.yaml": failed to create secrets.v1./master-user-data -n openshift-machine-api: namespaces "openshift-machine-api" not found
Jun 19 17:19:00 localhost bootkube.sh[5122]: "99_openshift-cluster-api_worker-user-data-secret.yaml": failed to create secrets.v1./worker-user-data -n openshift-machine-api: namespaces "openshift-machine-api" not found
Jun 19 17:19:00 localhost bootkube.sh[5122]: Created "99_openshift-cluster-api_master-user-data-secret.yaml" secrets.v1./master-user-data -n openshift-machine-api
Jun 19 17:19:00 localhost bootkube.sh[5122]: Created "99_openshift-cluster-api_worker-user-data-secret.yaml" secrets.v1./worker-user-data -n openshift-machine-api

... is from the excerpt...

Jun 19 17:17:03 localhost bootkube.sh[1314]: Starting cluster-bootstrap...
[...]
Jun 19 17:17:47 localhost bootkube.sh[5122]: [#187] failed to fetch discovery: Get "https://localhost:6443/api?timeout=32s": dial tcp [::1]:6443: connect: connection refused

... that things start to go wrong and finally the final excerpt keeps repeating itself until it reaches the lines...

Jun 19 17:19:00 localhost bootkube.sh[5122]: Created "99_openshift-cluster-api_master-user-data-secret.yaml" secrets.v1./master-user-data -n openshift-machine-api
Jun 19 17:19:00 localhost bootkube.sh[5122]: Created "99_openshift-cluster-api_worker-user-data-secret.yaml" secrets.v1./worker-user-data -n openshift-machine-api

Additionally I tried to access the URLs https://<BOOTSTRAP_LOCAL>:6443 , https://<BOOTSTRAP_IP>:6443 , https://<SERVICES_IP>:6443 and got the result below in all cases...

[core@localhost ~]$ curl -k https://127.0.0.1:6443
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {
    
  },
  "code": 403
}

. <BOOTSTRAP_LOCAL> - 127.0.0.1 ;
. <BOOTSTRAP_IP> - 10.3.0.4 ;
. <SERVICES_IP> - 10.3.0.14 .

Thanks! =D

3 replies

eduardolucioac Jun 19, 2021
Author

@vrutkovs and anyone who can help... =D

Please attach or link to log bundle

Another "log-bundle", but this time with the three master nodes running...

log-bundle-20210619161810.tar.gz

PLUS: I noticed that the masters nodes initially have the machine name defined as "okd_master_X" ("okd_master_1") , but after a while they restart and this name becomes "localhost". This is normal?

vrutkovs Jun 20, 2021
Maintainer

but after a while they restart and this name becomes "localhost". This is normal?

No, this means hostname is not set via DHCP/kernel args. See https://docs.okd.io/latest/installing/installing_bare_metal/installing-bare-metal.html#installation-network-user-infra_installing-bare-metal

eduardolucioac Jun 20, 2021
Author

Another "log-bundle", but this time with the three master nodes running..

log-bundle-20210619161810.tar.gz

@vrutkovs

log-bundle-20210619161810.tar.gz

Sorry for insisting, but did you notice anything strange in the logs I sent you? =D

I really don't know what else to do... It's very difficult to solve this problem...

Thanks! =D

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bootkube.service - Worker node does not terminate in bootstrap #704

{{title}}

Replies: 2 comments 4 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

bootkube.service - Worker node does not terminate in bootstrap #704

eduardolucioac Jun 19, 2021

Replies: 2 comments · 4 replies

vrutkovs Jun 19, 2021 Maintainer

eduardolucioac Jun 19, 2021 Author

eduardolucioac Jun 19, 2021 Author

eduardolucioac Jun 19, 2021 Author

vrutkovs Jun 20, 2021 Maintainer

eduardolucioac Jun 20, 2021 Author

eduardolucioac
Jun 19, 2021

Replies: 2 comments 4 replies

vrutkovs
Jun 19, 2021
Maintainer

eduardolucioac Jun 19, 2021
Author

eduardolucioac
Jun 19, 2021
Author

eduardolucioac Jun 19, 2021
Author

vrutkovs Jun 20, 2021
Maintainer

eduardolucioac Jun 20, 2021
Author