Worker machine does not reboot properly

[Diegunio]

In every environment I install via UPI, regardless of the OKD SCOS version (4.13-4.16), there is sometimes a situation where the worker machine does not boot up correctly after a reboot, and I need to restart it again from the hypervisor level, as there is no SSH access to it. Here’s the error that appears:

System is booting up. Unprivileged users are not permitted to log in yet. Please come back later. For technical details, see pam_nologin(8).

I have added an SSH key for the "core" user. I suspect that adding a key for the "root" user would allow me to access the host and restart the machine via SSH, but this still does not solve the issue of the machine not always booting up correctly.

Has anyone else encountered similar problems? I don't use too many MCs, so I think it's not due to the configuration of my environment.

[Goose29]

If the system is still in its boot phase, and is unable to get past boot stages / is broken due to a bad MachineConfig - which happens every now and then with bad configs. Since no root user is defined, and core user is locked out until the system is booted.

From my experience you would have to enter recovery mode, by catching the Grub menu, by removing console=ttyS0,115200n8 and adding single to the end of the linux kernel line. If manual changes are made to make the machine's filesystem, you would then have to update the MachineConfig to match the new state of the filesystem, and force the Machine Config Operator since it now thinks the ostree deployment is out-of-sync.

Which would look something like this:

oc debug node/[node_name]
chroot /host
rm /etc/machine-config-daemon/currentconfig
touch /run/machine-config-daemon-force

If the config state is entirely bonkers, then the rendered machine configs can be manipulated, and tricked into thinking its up-to-date.

Hope this helps :)