[OKD4.7] OKD version assignment to desired CoreOS Releases - CoreOS might not getting updated?

[devzeronull]

Hi all,

when taking a look at https://github.com/openshift/okd/releases you can see the desired Fedora CoreOS Version for each Release under Component Versions -> machine-os.

I parsed the Releases which brought me to:

Name: 4.7.0-0.okd-2021-06-04-191031
  machine-os 34.20210518.3 Fedora CoreOS

Name: 4.7.0-0.okd-2021-06-04-191031
  machine-os 34.20210518.3 Fedora CoreOS

Name: 4.7.0-0.okd-2021-05-22-050008
  machine-os 33.20210426.3 Fedora CoreOS

Name: 4.7.0-0.okd-2021-05-22-050008
  machine-os 33.20210426.3 Fedora CoreOS

Name: 4.7.0-0.okd-2021-04-24-103438
  machine-os 33.20210328.3 Fedora CoreOS

Name: 4.7.0-0.okd-2021-04-11-124433
  machine-os 34.20210406.10 Fedora CoreOS

Name: 4.7.0-0.okd-2021-03-28-152009
  machine-os 33.20210323.10 Fedora CoreOS

Name: 4.7.0-0.okd-2021-03-21-094146
Name: 4.7.0-0.okd-2021-03-07-090821
Name: 4.7.0-0.okd-2021-03-06-124908
Name: 4.7.0-0.okd-2021-02-25-144700
  machine-os 33.20210224.10 Fedora CoreOS

We are right now running OKD 4.7.0-0.okd-2021-04-11-124433 which should have machine-os "Fedora CoreOS 34.20210406.10".
But when looking at our nodes it says that we are running Fedora CoreOS Version 33.20210314.3.0 now (initially installed with 33.20210301.3.1 which got automatically updated).

I know about the OS Version problematic concerning inital cluster setups, but normally a running cluster should update its CoreOS version automatically at least up to the desired one (in our case "34.20210406.10"), or am I wrong?

Best regards

[vrutkovs]

We are right now running OKD 4.7.0-0.okd-2021-04-11-124433 which should have machine-os "Fedora CoreOS 34.20210406.10".
looking at our nodes it says that we are running Fedora CoreOS Version 33.20210314.3.0 now

There was a bug in labelling code, which mistakenly marked this release as containing FCOS 34.
This was fixed in the nightlies, and the next stable version says FCOS content was "downgraded" back to "33.20210328.3".

machine-os-content itself didn't switch to F34 and back, it remained 33.20210328.3

[devzeronull]

Hi, thank you for replying!
I made a mistake, initially to install we used CoreOS 33.20210217.3.0 as supposed - but anyways the problem stays...

The question would be now, why it didn't get updated to 33.20210328.3 and remained in "33.20210314.3.0" - is there a way to force CoreOS to update to this intended Version?

(Maybe this could also be a solution for our upgrade issues)

[vrutkovs]

The question would be now, why it didn't get updated to 33.20210328.3 and remained in "33.20210314.3.0" - is there a way to force CoreOS to update to this intended Version?

Check machine-config operator and machineconfigpool statuses - it should have updated all the nodes.
Users can't arbitrarily choose OS version to update from - all nodes are uniform and controlled by MCO

[devzeronull]

This is exactly our problem, since we are on a faulty release and CoreOS wasn't upgraded to the desired version automatically and now missing directories in the tree, the MCO throws an Error and stucks the upgrade procedure being degraded.

So the Workaround would be to upgrade the CoreOS Version to the desired one and then continue with it, or what do you think?

[vrutkovs]

upgrade the CoreOS Version

This cannot be done manually. Degraded MCO needs to be reported as a bug