[OKD-SCOS v4.16] Unable to provision UPI cluster on OpenStack

[jacksgt]

Hello,

thanks for all the hard work on getting the testing releases for OKD SCOS out :-)

I tried to bootstrap a cluster from https://origin-release.ci.openshift.org/releasestream/4-scos-next/release/4.16.0-0.okd-scos-2024-08-01-132038 (quay.io/okd/scos-release:4.16.0-0.okd-scos-2024-08-01-132038) with our usual UPI process on OpenStack private cloud.
The control plane nodes come up fine, but the worker nodes fail to join the cluster due to:

[ 4203.752942] ignition[804]: GET https://api-int.example.com:22623/config/standard: attempt #789
[ 4203.755077] ignition[804]: GET error: Get "https://api-int.example.com:22623/config/standard": dial tcp [1.2.3.4]:22623: connect: connection refused

Checking on the control plane nodes, the machine-config-server deployment seems to be missing:

$ oc -n openshift-machine-config-operator get pod
NAME                                                  READY   STATUS             RESTARTS         AGE
kube-rbac-proxy-crio-clu-jack-okd-scos-master-2x7w9   1/1     Running            3 (117m ago)     116m
kube-rbac-proxy-crio-clu-jack-okd-scos-master-cm9fb   1/1     Running            4                117m
kube-rbac-proxy-crio-clu-jack-okd-scos-master-g26hg   1/1     Running            3 (117m ago)     117m
machine-config-daemon-brxgn                           1/2     CrashLoopBackOff   27 (2m4s ago)    115m
machine-config-daemon-rggk7                           1/2     CrashLoopBackOff   32 (3m28s ago)   115m
machine-config-daemon-xwcvg                           1/2     CrashLoopBackOff   27 (2m5s ago)    115m
machine-config-operator-5bb45cf447-qk9tx              2/2     Running            1 (25m ago)      117m
$ oc -n openshift-machine-config-operator get deploy
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
machine-config-operator   1/1     1            1           117m

The MCO cluster operator is stuck at:

status:
  conditions:
  - lastTransitionTime: "2024-08-13T12:25:32Z"
    message: Working towards 4.16.0-0.okd-scos-2024-08-01-132038
    status: "True"
    type: Progressing
  - lastTransitionTime: "2024-08-13T12:35:35Z"
    message: 'Unable to apply 4.16.0-0.okd-scos-2024-08-01-132038: error during waitForDaemonsetRollout:
      [context deadline exceeded, daemonset machine-config-daemon is not ready. status:
      (desired: 3, updated: 3, ready: 0, unavailable: 3)]'
    reason: MachineConfigDaemonFailed
    status: "True"
    type: Degraded

So the MCO is refusing to make progress because the machine-config-daemonset is unhealthy, therefore it has not deployed the machine-config-server DS.

The machine-config-daemonset pods are in CrashLoopBackOff and are logging:

oc -n openshift-machine-config-operator logs machine-config-daemon-xwcvg
Defaulted container "machine-config-daemon" out of: machine-config-daemon, kube-rbac-proxy
I0813 14:18:49.926288   96401 start.go:68] Version: machine-config-daemon-4.6.0-202006240615.p0-2819-ga4756652-dirty (a4756652d07e380d7c6176af382f756d5c48281f)
I0813 14:18:49.926344   96401 update.go:2618] Running: mount --rbind /run/secrets /rootfs/run/secrets
I0813 14:18:49.928361   96401 update.go:2618] Running: mount --rbind /usr/bin /rootfs/run/machine-config-daemon-bin
I0813 14:18:49.931021   96401 daemon.go:517] using appropriate binary for source=rhel-9 target=rhel-9
I0813 14:18:49.975724   96401 daemon.go:570] Invoking re-exec /run/bin/machine-config-daemon
I0813 14:18:50.008591   96401 start.go:68] Version: machine-config-daemon-4.6.0-202006240615.p0-2819-ga4756652-dirty (a4756652d07e380d7c6176af382f756d5c48281f)
E0813 14:18:50.008755   96401 rpm-ostree.go:276] Merged secret file does not exist; defaulting to cluster pull secret
I0813 14:18:50.008779   96401 rpm-ostree.go:263] Linking ostree authfile to /var/lib/kubelet/config.json
F0813 14:18:50.086251   96401 start.go:106] Failed to initialize single run daemon: error reading osImageURL from rpm-ostree: exit status 1

rpm-ostree is unhappy because:

# rpm-ostree status
A dependency job for rpm-ostreed.service failed. See 'journalctl -xe' for details.
○ rpm-ostreed.service - rpm-ostree System Management Daemon
     Loaded: loaded (/usr/lib/systemd/system/rpm-ostreed.service; static)
    Drop-In: /etc/systemd/system/rpm-ostreed.service.d
             └─10-mco-default-env.conf
             /run/systemd/system/rpm-ostreed.service.d
             └─bug2111817.conf
             /etc/systemd/system/rpm-ostreed.service.d
             └─mco-controlplane-nice.conf
     Active: inactive (dead)
       Docs: man:rpm-ostree(1)

Aug 13 14:12:24 clu-jack-okd-scos-master-cm9fb systemd[1]: Dependency failed for rpm-ostree System Management Daemon.
Aug 13 14:12:24 clu-jack-okd-scos-master-cm9fb systemd[1]: rpm-ostreed.service: Job rpm-ostreed.service/start failed with result 'dependency'.
Aug 13 14:17:27 clu-jack-okd-scos-master-cm9fb systemd[1]: Dependency failed for rpm-ostree System Management Daemon.
Aug 13 14:17:27 clu-jack-okd-scos-master-cm9fb systemd[1]: rpm-ostreed.service: Job rpm-ostreed.service/start failed with result 'dependency'.
Aug 13 14:22:37 clu-jack-okd-scos-master-cm9fb systemd[1]: Dependency failed for rpm-ostree System Management Daemon.
Aug 13 14:22:37 clu-jack-okd-scos-master-cm9fb systemd[1]: rpm-ostreed.service: Job rpm-ostreed.service/start failed with result 'dependency'.
Aug 13 14:23:00 clu-jack-okd-scos-master-cm9fb systemd[1]: Dependency failed for rpm-ostree System Management Daemon.
Aug 13 14:23:00 clu-jack-okd-scos-master-cm9fb systemd[1]: rpm-ostreed.service: Job rpm-ostreed.service/start failed with result 'dependency'.
Aug 13 14:23:07 clu-jack-okd-scos-master-cm9fb systemd[1]: Dependency failed for rpm-ostree System Management Daemon.
Aug 13 14:23:07 clu-jack-okd-scos-master-cm9fb systemd[1]: rpm-ostreed.service: Job rpm-ostreed.service/start failed with result 'dependency'.
error: Loading sysroot: exit status: 1

The offending dependency seems to be systemd-fsck:

Aug 13 14:23:00 clu-jack-okd-scos-master-cm9fb systemd[1]: Starting File System Check on /dev/disk/by-uuid/c4d3a448-f7f7-4bc6-8887-d034fbd667c2...
░░ Subject: A start job for unit systemd-fsck@dev-disk-by\x2duuid-c4d3a448\x2df7f7\x2d4bc6\x2d8887\x2dd034fbd667c2.service has begun execution
░░ Defined-By: systemd
░░ Support: https://access.redhat.com/support
░░ 
░░ A start job for unit systemd-fsck@dev-disk-by\x2duuid-c4d3a448\x2df7f7\x2d4bc6\x2d8887\x2dd034fbd667c2.service has begun execution.
░░ 
░░ The job identifier is 4863.
Aug 13 14:23:00 clu-jack-okd-scos-master-cm9fb systemd-fsck[24560]: /dev/vda3 has unsupported feature(s): FEATURE_C12
Aug 13 14:23:00 clu-jack-okd-scos-master-cm9fb systemd-fsck[24560]: e2fsck: Get a newer version of e2fsck!
Aug 13 14:23:00 clu-jack-okd-scos-master-cm9fb systemd-fsck[24560]: boot: ********** WARNING: Filesystem still has errors **********
Aug 13 14:23:00 clu-jack-okd-scos-master-cm9fb systemd-fsck[24558]: fsck failed with exit status 12.
Aug 13 14:23:00 clu-jack-okd-scos-master-cm9fb systemd[1]: systemd-fsck@dev-disk-by\x2duuid-c4d3a448\x2df7f7\x2d4bc6\x2d8887\x2dd034fbd667c2.service: Main process exited, code=exited, status=1/FAILURE

So the final error is:

/dev/vda3 has unsupported feature(s): FEATURE_C12
e2fsck: Get a newer version of e2fsck!
boot: ********** WARNING: Filesystem still has errors **********
fsck failed with exit status 12.

Could this be caused by the fact that node was originally booted from fedora-coreos-39.20240210.3.0-openstack.x86_64 and then pivoted to SCOS9 (417.9.202407210037-0)?

I'm grateful for any additional hints for troubleshooting.

[SriRamanujam]

I think you may be seeing the same issue as in this discussion

[jacksgt]

Thanks for the link, it sure sounds the like the same issue!

However in my case this happens in a fresh install, not a FCOS -> SCOS migration.

[jacksgt]

It turned out that I was using a slightly wrong boot image (fedora-coreos-39.20240210.3.0-openstack.x86_64 instead of fedora-coreos-39.20231101.3.0-openstack.x86_64).
With the correct boot image (from openshift-install coreos print-stream-json | jq -r '.architectures.x86_64.artifacts.openstack.formats["qcow2.xz"]') it works!