Recent containers storage.conf breaking RHCOS promotion

[cgwalters]

I'm not sure this is a MCO (containerruntimeconfig) bug but it seems likely. Recent RHCOS 4.3 promotions are dying with E1210 16:15:51.105286 11181 daemon.go:1350] content mismatch for file /etc/containers/storage.conf::

https://storage.googleapis.com/origin-ci-test/logs/promote-release-openshift-machine-os-content-e2e-aws-4.3/1012/artifacts/e2e-aws/pods/openshift-machine-config-operator_machine-config-daemon-29fqh_machine-config-daemon.log

This seems to have been provoked with a change to the default in the containers-common package. I don't understand how this could break anything though.

Hum...unless - if somehow the cconfigc¹ isn't running at bootstrap time, and we were just happening to rely on the contents being the same as the RPM package. Then, our generated Ignition wouldn't include the changes, but the MCD would expect to find them. But hmm...we md5sum the generated Ignition, so that seems unlikely.

¹ for short, also it's somehow hypnotic spelled that way

[vrutkovs]

MCO uses static template when containerruntimeconfig is applied, including it in the Ignition file applied to the machine before machine-os-content is applied. This would break if storage.conf is being changed in a newer RPM in machine-os-content - MCD won't be able to sync configs.

Could MCD peek into the staged deployment before the reboot, change particular settings and update Ignition?

[cgwalters]

But in the current code, the templated config should override whatever the RPM ships.

[cgwalters]

Now, it would obviously be a lot nicer if in fact e.g. we just merged any changes we wanted to make with the actual defaults from the RPM, but that would be a rewrite of the code.

(Really what we want is a drop-in containers.conf.d style thing)

[cgwalters]

Filed #1320 as a cleanup while looking at this but also as a CI test canary.

[cgwalters]

Ahhh wait, I think I know the problem! If the templated config is identical to the version shipped in the package, then ostree assumes that the admin hasn't modified it, and will replace it with the new default version.

If @umohnani8 based the copy in the MCO repo on the then-default package configuration, that would entirely explain this.

[haircommander]

If @umohnani8 based the copy in the MCO repo on the then-default package configuration, that would entirely explain this.

I remember the original template was straight from the RPM at the time

[vrutkovs]

Ah, so its ostree and MCD not playing well together :/

[vrutkovs]

Perhaps its worth adding smth like "# DO NOT MODIFY This file is controlled by MCO" to all MCD-controlled files to ensure this won't occur again?