Declarative layering builder/interface

[cgwalters]

Splitting this out from https://github.com/coreos/enhancements/blob/main/os/coreos-layering.md

Basically this thread is about how we offer a declarative layered coreos build interface that constrains input as much as possible to flows that we have tested.

For example with a Dockerfile today, one can invoke e.g. RUN useradd foo but such a thing right now deeply conflicts with the user management flows of rpm-ostree and Ignition and would need nontrivial work to support. What we need to do for now for local user management is include it in Ignition, and have Ignition run it on firstboot as happens normally today. Which in turn, means it can't be part of the ignition rendered into a container image, but needs to be part of the Ignition config provided to the machine on firstboot.

OK, the digression on useradd aside: I think what we should try to ship for this is basically something that can take a butane configuration and build a container image from it.

This would build on coreos/ignition#1285 - and really the biggest gap is having a declarative package install interface. Of course, rpm-ostree already has one of those but I don't think we should expose the full treefile; about half of those flags only apply to base images (e.g. boot-location) or would be messy to implement correctly install-langs. To start really, we could support a tiny subset of that yaml which allows specifying packages. Perhaps we could extend it with package-repos. Some people are definitely going to want lockfiles.

For the butane package bits, see #681
(And a big bonus would be if this same stanza worked for both container builds and as part of the per-machine config!)

So here's a strawman:

$ cat config.bu
variant: fcos
version: 1.10.0
stream: stable  # Expands to equivalent of `FROM quay.io/fedora/coreos:stable` for container builds
storage:
  files:
    - path: /etc/usbguard.conf
      contents:
        local: usbguard.conf
      mode: 0644
packages:
  - usbguard
$ coreos-derivation-builder -i config.bu -o containers-storage:localhost/my-fcos-with-usbguard
...
$ podman run --rm -ti localhost/my-fcos-with-usbguard
# rpm -q usbguard
...
# usbguard-1.2.5.fc37
# <ctrl-d>
$ coreos-derivation-builder -i config.bu -o containers-storage:localhost/my-fcos-with-usbguard
No changes in input.
$

Notice here a big benefit is we can carry forward the "change detection" that rpm-ostree has today for compose tree. This is coreos/rpm-ostree#3238

Supported butane inputs

We should error out on input butane stanzas like storage - things like LUKS and RAID need to be set up by the Ignition provided on firstboot.

[cgwalters]

There is an alternative path here where we still support being mostly declarative, but it happens inside a Dockerfile build. The benefit of this is we work anywhere Dockerfile works (e.g. quay.io builds, default OCP builds etc.) but we lose things like change detection. This touches on containers/podman#12605

OK, well actually this was written out before in https://github.com/coreos/enhancements/blob/main/os/coreos-layering.md#butane-as-a-declarative-input-format-for-layering
but it needs to be cleaned up a bit, and we basically need a new tool that also knows how to run rpm-ostree install for packages etc.

Or, there's a potential here to say let's not have a new tool, let's teach rpm-ostree about ignition and butane. I am...not opposed to that but it's a notable increase in scope, and raises questions about the relationship of ignition/butane and the other systems using rpm-ostree but not ignition such as current Fedora Silverblue/IoT etc. This blending of packages and configuration state would definitely be more NixOS like though.

[mkenigs]

Not sure if I'm fully tracking with everything you're describing, but it feels like the direction you're explaining this is take something declarative and build a container layer as an output. To me it seems like that discards some of the benefits of using ostree.

If we took the route of teaching rpm-ostree about Butane, we could support container layers while using more intelligent packaging in the background. Which parts of Butane would rpm-ostree need to know? Files, systemd units, and users? (it already understands kernel arguments, right?) And everything else would be on firstboot?

The strawman I was thinking of was teach treefiles everything we need after firstboot, and then make them composable. That way if a user wants to add a single rpm package, they just write a treefile with that package, and rpm-ostree can handle that intelligently. Rather than the user building a container layer containing the rpm which rpm-ostree can't touch. If we still want to provide the user with Dockerfile as an interface, we could still allow that as something layered on top. The user could still install rpms that way, it would just be less efficient. That could also give us a way to support files and users before teaching rpm-ostree about them.

[cgwalters]

This is a super interesting topic for sure. So one thing you mentioned on internal chat is actually we do have ostree-layers and ostree-override-layers in rpm-ostree which allow injecting arbitrary non-RPM content.

This simplifies things for coreos-derivation-builder because it could use e.g. butane and ignition-liveapply and then writes the generated data into an ostree commit. This would be passed to rpm-ostree compose tree which already has all the build and change detection logic.

If we took the route of teaching rpm-ostree about Butane,

Hmm. There are definitely benefits to this, among them that we're not introducing a new CLI verb (and a new separate git repo with its own CI, releases, versioning etc.).

I guess we could make this a "soft" dependency, i.e. the rpm-ostree.rpm wouldn't actually Requires: ignition but perhaps just Suggests: or even just failing at runtime if you specify a butane configuration but you don't have /usr/bin/butane. OTOH again as noted above there are other projects which use rpm-ostree and not ignition, and I don't want them to feel "second class" in any way. The initial goal of rpm-ostree was to just handle OS updates, not configuration. There are people successfully using rpm-ostree in combination with e.g. Ansible and similar tools.

There's also overlap here with osbuild manifests and it's also possible to involve kickstarts.

OTOH, I've thought in the past about writing a kickstart ➡️ ignition (or butane) translator - it wouldn't be hard to get coverage of the 80% case.

And ultimately, rpm-ostree still would maintain ostree-layers which can be ostree commits which can be generated by anything as an un-opinonated API.

I'm not sure.

[mkenigs]

The initial goal of rpm-ostree was to just handle OS updates, not configuration. There are people successfully using rpm-ostree in combination with e.g. Ansible and similar tools.

Either the MCO needs a tool to handle every part of OS state, or it has to be that tool. We already use rpm-ostree, and our users are already familiar with Dockerfiles. So a few of our options (ignoring Ignition firstboot stuff for now) are:

1. The MCO handles OS state which could mean either of:
   1. Stitch different tools together with the MCO, eg OS state is a tuple of (Ignition/MachineConfig, machine-os-content, container image layers). Afaict, the biggest flaw of the current MCO is that we try to do this, but we don't handle Ignition/MachineConfig in a way we can (cleanly) revert, so we create bugs where we lose track of OS state. And we can't pre-build images. I think we've decided that handling this well would require something like option 2 or 3
   2. Define a new format for OS state (I'm assuming that would be too much effort)
2. OS state is encapsulated in container image layers. I don't think layering binary blobs is a very intelligent way of handling OS state, because they're opaque and can't be reused efficiently. So I don't think this is a good option
3. rpm-ostree manages all of OS state, eg the OS state is fully described by a treefile (which could contain a reference to a Dockerfile)

[cgwalters]

Either the MCO needs a tool to handle every part of OS state, or it has to be that tool.

Yes, this is a great way to say it. This is an important debate to have.

OK so today, the MCO is that tool - the rendered machineconfig combines the OS binaries with the configuration. Except for anything which came from the pointer ignition which we need to support for per-node state. (But, right now we are not trying to support "day 2" changes to this state either)

OS state is encapsulated in container image layers. I don't think layering binary blobs is a very intelligent way of handling OS state, because they're opaque and can't be reused efficiently.

I think the goal here though actually is that we can say that the OS state can be described by a container image, and not a rendered machineconfig. I'd say a container image is actually much less opaque than the rendered machineconfig today, because tons and tons of tools know how to introspect and scan them; but machineconfig CRDs are specific to OpenShift 4.

rpm-ostree manages all of OS state, eg the OS state is fully described by a treefile (which could contain a reference to a Dockerfile)

The thing is the treefile today is not at all designed to capture configuration (that's where Ignition is). Where we've landed with coreos-assembler/FCOS/RHCOS is that it supports injecting an overlay.d directory from the source git repository. But there are interesting consequences to this - among them that all those systemd units are not tracked by the RPM database.

[mkenigs]

I think the goal here though actually is that we can say that the OS state can be described by a container image, and not a rendered machineconfig. I'd say a container image is actually much less opaque than the rendered machineconfig today, because tons and tons of tools know how to introspect and scan them; but machineconfig CRDs are specific to OpenShift 4.

I more meant as compared to something like rpm-ostree. Having a set of RPMs seems less opaque than having multiple RPMs stored in one layer which you have to introspect. Eg the current approach for deduplication with container layers is loop through all the files in the layer (introspection) rather than just having an RPM database. Or the approach for overriding packages is overlay them at runtime (that's almost runtime introspection), which seems much more inefficient and opaque than just saying you want a different version of an RPM.

[cgwalters]

Actually though, for the MCO (and since this is a general FCOS issue, also outside of it) I'd say a huge part of the entire idea here is that from a per-machine perspective, all that happens in order to apply an update is that we pull and update to a container image and that's it; as little scripting and per-node state outside of that as possible.

(Another way to say this is (as you know but it's worth restating) that this approach will fix openshift/machine-config-operator#1190 which IMO will be huge in and of itself)

So this issue is really more about an opinionated, efficient, supportable way to build that image.

This of course does tie somewhat into the per-machine state; for example, if part of our opinionated model is that there's a nice declarative file set stored in git (again, much like fedora-coreos-config) then perhaps we can teach rpm-ostree status to render the container metadata for that in a nice way. (But I'd also like rpm-ostree to continue to support multiple ways to build and ship content in the same way container images do, so such a thing should at least be somewhat generic)

Anyways though, as far as this whole thing; what I'm thinking is:

• Add support for package installs to butane
• The derivation builder supports generating a new container image from an inline butane config, as well as a butane config stored in a git repository

I more meant as compared to something like rpm-ostree. Having a set of RPMs seems less opaque than having multiple RPMs stored in one layer which you have to introspect. Eg the current approach for deduplication with container layers is loop through all the files in the layer (introspection) rather than just having an RPM database. Or the approach for overriding packages is overlay them at runtime (that's almost runtime introspection), which seems much more inefficient and opaque than just saying you want a different version of an RPM.

I don't quite understand what you mean here - we can maybe pick this up in a real-time conversation and re-summarize here. Actually this whole thing would be a great topic for the next FCOS community meeting.

[jlebon]

Anyways though, as far as this whole thing; what I'm thinking is:

• Add support for package installs to butane

This SGTM, and to be clear, re.

For the butane package bits, see #681
(And a big bonus would be if this same stanza worked for both container builds and as part of the per-machine config!)

To me, this is exactly the kind of thing that makes using Butane more appealing for this. It means users only have to keep understanding Butane. We can just say e.g. "you can now apply your Butane config before deploying".

The flip side to this though is that if some of the Butane sugar knobs we want add in the future are only valid for CoreOS layering and not first boot, it'll be really confusing. So... ideally everything CoreOS layering supports would be a subset of what first boot supports (obviously partitioning and e.g. kargs means they'll never be at parity).

[cgwalters]

One conclusion here is that we will likely want to compile butane into rpm-ostree treefiles, so it can be the build engine. This will allow all the e.g. change detection to work. (OTOH, treefiles are weak on non-rpm content)

It'd be good to clarify what the MCO's role is - specifically does the MCO take butane as input - do we make it a CRD? Short term may be more "glue", long term may defer to rpm-ostree/CoreOS?

Once we add package installs to butane, the overlap with treefiles will actually get a bit larger, but that will help us understand whether butane is the right interface.

[mkenigs]

From discussion, the primary reason given to use Butane was that it's a better interface for users since it doesn't have some of the knobs that treefiles have.

Some of my own thoughts: while there's some distinction between what treefiles and Butane do, because there's some overlap in what they do (eg either could apply files or systemd units), it doesn't seem technically clear to me right now exactly what the distinction is. Not sure if this is correct, but right now it seems like the distinction is more that treefiles are a tool for us to build an image, and Butane is a tool for users to apply configuration.

That's making me wonder about some of the questions I have about the MCO's role/goals.

Is the MCO an operator to apply Red Hat supported options to nodes? In that case it makes sense that treefiles are a tool for us to build images, and Butane is a tool for users without all knobs enabled.

Is the MCO an operator that glues Kubernetes + Linux OS's? Then the MCO would need to support a more complete Spec for Linux nodes, and we could just disable anything we don't want to support. That would feel even more appropriate if we were downstream from OKD.

Maybe there's a 3rd option where the MCO glues Kubernetes + desirable options for Linux nodes, in which Butane would also make a lot of sense, but I don't have a grasp of why that would be the case.

[mkenigs]

@jkyros in case you aren't following this issue

[jlebon]

Not sure if this is correct, but right now it seems like the distinction is more that treefiles are a tool for us to build an image, and Butane is a tool for users to apply configuration.

This is correct. But you're right that there's definitely a lot of overlap conceptually between the two in the context of CoreOS layering. To some extent, either could work (or even a third separate DSL) and there isn't a clearly "right" or "wrong" choice. They both would require some amount of work to adapt to the use case here. As mentioned earlier, my vote goes to Butane for various reasons, but as Colin said, at this point, we need to try, learn, and iterate.