Clusters cannot be upgraded

[gerijani]

This was the first time I tried to upgrade. First, on console Administration > Cluster Settings UPDATE STATUS showed Updates Available to 4.7 without choosing Channel. After I changed to stable-4.6, update possibility dissapeared and I can not update between minor versions.

$oc get clusterversion
NAME      VERSION                         AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.6.0-0.okd-2021-02-14-205305   True        False         75d     Cluster version is 4.6.0-0.okd-2021-02-14-205305

But using cli show problem:

$oc adm upgrade
Cluster version is 4.6.0-0.okd-2021-02-14-205305

Upgradeable=False

  Reason: ClusterOperatorsNotUpgradeable
  Message: Multiple cluster operators cannot be upgraded between minor versions:
* Cluster operator etcd cannot be upgraded between minor versions:
UnsupportedConfigOverrides_UnsupportedConfigOverridesSet:
UnsupportedConfigOverridesUpgradeable: setting: 
[useUnsupportedUnsafeNonHANonProductionUnstableEtcd useUnsupportedUnsafeNonHANonProductionUnstableOAuthServer]
* Cluster operator authentication cannot be upgraded between minor versions: 
UnsupportedConfigOverrides_UnsupportedConfigOverridesSet: 
UnsupportedConfigOverridesUpgradeable: setting: 
[useUnsupportedUnsafeNonHANonProductionUnstableOAuthServer]

Cluster operators does not show any degradation:

$ oc get co
NAME                                       VERSION                         AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.6.0-0.okd-2021-02-14-205305   True        False         False      99m
cloud-credential                           4.6.0-0.okd-2021-02-14-205305   True        False         False      75d
cluster-autoscaler                         4.6.0-0.okd-2021-02-14-205305   True        False         False      75d
config-operator                            4.6.0-0.okd-2021-02-14-205305   True        False         False      75d
console                                    4.6.0-0.okd-2021-02-14-205305   True        False         False      106m
csi-snapshot-controller                    4.6.0-0.okd-2021-02-14-205305   True        False         False      11d
dns                                        4.6.0-0.okd-2021-02-14-205305   True        False         False      104m
etcd                                       4.6.0-0.okd-2021-02-14-205305   True        False         False      75d
image-registry                             4.6.0-0.okd-2021-02-14-205305   True        False         False      75d
ingress                                    4.6.0-0.okd-2021-02-14-205305   True        False         False      75d
insights                                   4.6.0-0.okd-2021-02-14-205305   True        False         False      75d
kube-apiserver                             4.6.0-0.okd-2021-02-14-205305   True        False         False      75d
kube-controller-manager                    4.6.0-0.okd-2021-02-14-205305   True        False         False      75d
kube-scheduler                             4.6.0-0.okd-2021-02-14-205305   True        False         False      75d
kube-storage-version-migrator              4.6.0-0.okd-2021-02-14-205305   True        False         False      102m
machine-api                                4.6.0-0.okd-2021-02-14-205305   True        False         False      75d
machine-approver                           4.6.0-0.okd-2021-02-14-205305   True        False         False      75d
machine-config                             4.6.0-0.okd-2021-02-14-205305   True        False         False      75d
marketplace                                4.6.0-0.okd-2021-02-14-205305   True        False         False      100m
monitoring                                 4.6.0-0.okd-2021-02-14-205305   True        False         False      137m
network                                    4.6.0-0.okd-2021-02-14-205305   True        False         False      75d
node-tuning                                4.6.0-0.okd-2021-02-14-205305   True        False         False      149m
openshift-apiserver                        4.6.0-0.okd-2021-02-14-205305   True        False         False      108m
openshift-controller-manager               4.6.0-0.okd-2021-02-14-205305   True        False         False      103m
openshift-samples                          4.6.0-0.okd-2021-02-14-205305   True        False         False      75d
operator-lifecycle-manager                 4.6.0-0.okd-2021-02-14-205305   True        False         False      75d
operator-lifecycle-manager-catalog         4.6.0-0.okd-2021-02-14-205305   True        False         False      75d
operator-lifecycle-manager-packageserver   4.6.0-0.okd-2021-02-14-205305   True        False         False      98m
service-ca                                 4.6.0-0.okd-2021-02-14-205305   True        False         False      75d
storage                                    4.6.0-0.okd-2021-02-14-205305   True        False         False      75d

If I tried to force channel 4.7 with " oc patch clusterversion version --type="merge" -p '{"spec":{"channel":"stable-4.7"}}' " the upgrade still failing with

[vrutkovs]

Single node masters can't be upgraded just yet via console. This would be resolved in 4.8.

The workaround:

• Make etcd backup. Single master upgrades are dangerous
• Add override for etcd quorum guard in ClusterVersion:

  overrides:
    - group: apps/v1
      kind: Deployment
      name: etcd-quorum-guard
      namespace: openshift-etcd
      unmanaged: true

• Scale it back to 1 replica
• Force upgrade by setting .spec.desired.force: true in ClusterVersion
• When etcd operator upgrades, remove override
• When upgrade reaches the stage when machine-config-operator is updated run https://github.com/vrutkovs/okd-installer/blob/master/manifests/singlenode/upgrade-master.sh on the node via ssh.
  This script must be terminated when it prints "rpm-ostree upgrade" command.
  Copy the resulting command and run it manually:

rpm-ostree rebase --experimental /run/mco-machine-os-content/<some folder>/srv/repo:<some hash> --custom-origin-url pivot://quay.io/openshift/okd-content@<another hash> --custom-origin-description "Managed by machine-config-operator"

This would reboot the machine, when it restarts it should be back. If that didn't work, boot from previous rpm-ostree deployment and restore from etcd backup

[gerijani]

It's a 3 master node cluster:

$ oc get nodes
NAME                                   STATUS   ROLES    AGE     VERSION
perf2-master.perf2.local       Ready    master   75d     v1.19.2+f173eb4-1049
perf2-slave.perf2.local        Ready    master   75d     v1.19.2+f173eb4-1049
perf2-slave2.perf2.local       Ready    master   3h52m   v1.19.2+f173eb4-1049
perf2-testworker.perf2.local   Ready    worker   75d     v1.19.2+f173eb4-1049

[vrutkovs]

Why do you have unsupportedNonHA settings? Did you scale it up from 1 or two masters?

[gerijani]

From 2, but I tried even :

oc patch etcd cluster -p='{"spec": {"unsupportedConfigOverrides": {"useUnsupportedUnsafeNonHANonProductionUnstableEtcd": true }}}' --type=merge
and for "useUnsupportedUnsafeNonHANonProductionUnstableOAuthServer".

Then restarted all pods with

for I in $(oc get ns -o jsonpath='{range .items[*]} {.metadata.name}{"\n"} {end}'); \
       do oc delete pods --all -n $I; \
       sleep 1; \
done

No change.

[vrutkovs]

You can't install a cluster with 2 masters (that would be a single master install with additional node anyway).

Since you now have three master nodes check that etcd has 3 members and remove the unsupported configs?

[gerijani]

Why do you have unsupportedNonHA settings? Did you scale it up from 1 or two masters?

Yes, that gave me actually the idea to remove (oc edit ...) these settings from etcd cluster and authentications.operator.openshift.io cluster and it did the trick.
Solved.