Content mismatch for file \"/etc/pki/ca-trust/source/anchors/openshift-config-user-ca-bundle.crt\"

[K1kc4]

Describe the bug

Node master1.okd4-test.xxx.org is reporting: "unexpected on-disk state validating against rendered-master-8f0b3e01aefd81fddcce641a03aefd44: content mismatch for file \"/etc/pki/ca-trust/source/anchors/openshift-config-user-ca-bundle.crt\"", Node master2.okd4-test.xxx.org is reporting: "unexpected on-disk state validating against rendered-master-8f0b3e01aefd81fddcce641a03aefd44: content mismatch for file \"/etc/pki/ca-trust/source/anchors/openshift-config-user-ca-bundle.crt\"", Node master3.okd4-test.xxx.org is reporting: "unexpected on-disk state validating against rendered-master-8f0b3e01aefd81fddcce641a03aefd44: content mismatch for file \"/etc/pki/ca-trust/source/anchors/openshift-config-user-ca-bundle.crt\""

Version
4.13.0-0.okd-2023-09-30-084937 to 4.14.0-0.okd-2023-10-28-073550

Fixx
https://access.redhat.com/solutions/5099331

I had to do this twice on master1, before it actually started working.

[K1kc4]

This only happens when updating to 4.14..I updated 4.13.0-0.okd-2023-09-30-084937 to 4.13.0-0.okd-2023-10-28-065448 and it went smooth.
Then again i tried from 4.13.0-0.okd-2023-10-28-065448 to 4.14.0-0.okd-2023-11-14-101924 and i was getting the same error.

Mco logs:

E1115 14:00:14.121333  565938 daemon.go:756] Preflight config drift check failed: unexpected on-disk state validating against rendered-master-f892df52ba58d94ee4050012c9897718: content mismatch for file "/etc/pki/ca-trust/source/anchors/openshift-config-user-ca-bundle.crt"
E1115 14:00:14.121351  565938 writer.go:226] Marking Degraded due to: unexpected on-disk state validating against rendered-master-f892df52ba58d94ee4050012c9897718: content mismatch for file "/etc/pki/ca-trust/source/anchors/openshift-config-user-ca-bundle.crt"
I1115 14:01:14.165941  565938 daemon.go:1519] File "/etc/machine-config-daemon/currentimage" does not yet exist, defaulting to empty value
W1115 14:01:14.166043  565938 daemon.go:2073] current+desiredConfig is rendered-master-f892df52ba58d94ee4050012c9897718 but state is Degraded
I1115 14:01:14.169239  565938 daemon.go:1519] File "/etc/machine-config-daemon/currentimage" does not yet exist, defaulting to empty value
I1115 14:01:14.231634  565938 rpm-ostree.go:307] Running captured: rpm-ostree kargs
E1115 14:01:14.320556  565938 on_disk_validation.go:237] content mismatch for file "/etc/pki/ca-trust/source/anchors/openshift-config-user-ca-bundle.crt" (-want +got):
  bytes.Join({
        ... // 3889 identical bytes
        "/lThDYjTj7XceA/LDBZVHA7qgfq8VIq6\nEMEV07dr1cHzUs6vDktf7FiC9KL/Yy7",
        "21MXQWGy/PMs27gikf4AUqKEA5EohLL4/\nO20=\n-----END CERTIFICATE-----",
-       "\n",
  }, "")
E1115 14:01:14.320619  565938 daemon.go:756] Preflight config drift check failed: unexpected on-disk state validating against rendered-master-f892df52ba58d94ee4050012c9897718: content mismatch for file "/etc/pki/ca-trust/source/anchors/openshift-config-user-ca-bundle.crt"
E1115 14:01:14.320636  565938 writer.go:226] Marking Degraded due to: unexpected on-disk state validating against rendered-master-f892df52ba58d94ee4050012c9897718: content mismatch for file "/etc/pki/ca-trust/source/anchors/openshift-config-user-ca-bundle.crt"

I also tried manually adding \n and deleting \n on all the nodes, i was comparing all the files, and there was no difference between any of them..

After forcing Machine Config Daemon on all the nodes it continues the update, but sometimes it even requires me to do two time on the same node..

[K1kc4]

It still happens, tried with 4.14-12.01 too.
This must be a bug.
This is photo of /etc/pki/ca-trust/source/anchors/openshift-config-user-ca-bundle.crt BEFORE update:

This photo is when cluster update was on 97% and new Machine-Config-Operator was deployed:

It changed the cert, and there is \n missing. If i add \n manually, error is gone, node is drained/cordoned and thats it, it doesnt update the node on new machine-config...

[K1kc4]

Fixed with 4.14.0-0.okd-2024-01-06-084517 version.