remote config via configmap and secrets

[cgwalters]

This overlaps with #7 some.

Here, the basic idea is something like:

bootc config add [--root=/etc] https://examplecorp.com/config.yml

(OR with support for OCI Artifacts we support bootc config add [--root=/etc] registry:quay.io/examplecorp/config-server-base:latest)

Where config.yml is a standard Kubernetes ConfigMap. By default, we "mount" the keys to /etc. Then, bootc upgrade looks for updates to all provided configmaps - if any change, it triggers the same upgrade logic as the base image.

We also fetch and handle secret objects in the same way. It'd be cool though to support something like handling encrypted secrets (and configmaps) which need to be decrypted via a key (which could be in a TPM or so).

We also need to think carefully about file permissions; mode 0644 for all configmap files and 0600 for secrets may make sense. In addition we could support special annotations to override these.

(This should also work to be invoked immediately after bootc install to have it ready on the first boot, i.e. we also have a --root argument or so)

[cgwalters]

To flesh this out a bit more, bootc upgrade would check for updates to these things too.

Also, another variant of this is that we should support fetching all this stuff from the cloud user data too.

[cgwalters]

So this design would work in an obvious way outside of Kubernetes - we're using a standard API type, but we're not depending on an API server or kubelet, etc.

But, in a Kubernetes context, it would make total sense to fetch this data from the API server instead of just running http. (And potentially use proper watches to react to events instead of polling). I'm thinking we probably want to make the mechanism to fetch this data pluggable; ultimately it might be that there's /var/lib/bootc/config.d and a privileged container image could e.g. use kubelet's credentials to fetch from the API server instead, and just hand things down to bootc.

This intersects with openshift/machine-config-operator#3327 too.

[cgwalters]

This intersects with openshift/machine-config-operator#3327 too.

A hugely interesting topic here of course is whether we should have a declarative way to set this stuff up. Today for example we have an imperative interface in bootc switch to specify the root image.

But...following the kubelet "static pod" model, and building on the pod analogy above, perhaps we have
/etc/bootc/host.yaml which is a subset of the pod spec, and allows declaratively writing the attached configmaps.

(This topic also relates to coreos/rpm-ostree#2326 )

[cgwalters]

It may also be interesting here to try to support something like an annotation which says the configmap changes should be live-applied by default - in the general case we'd probably need some sort of "post-apply hook" to ask affected services to reload. Or, such services could use inotify of course (at the risk of seeing partial states).

[cgwalters]

To implement this I'm thinking:

• Add a kubernetes feature (on by default)
• Use the https://docs.rs/k8s-openapi/latest/k8s_openapi/ crate
• Support basic fetch via http, and investigate OCI artifact support (going to need to be driven into skopeo most likely)
• Write configmaps as ostree commits (map name to ref, etc.); although we may want to actually split into "metadata" and "contents"; metadata changes to the configmap shouldn't appear as changes probably, except things like the potential update semantics
• Create merged commit by applying configmap commits into /usr/etc to start - and here we have the interesting question of handling conflicts (i.e. what happens if my configmap has /etc/sudoers and the host has that file too? I'd probably vote that we error by default and require an annotation to overwrite?)
• bootc update looks for changes to base and all configmaps

Now...what many people will want is a read-only /etc; if we have that, it becomes logically easier to have each configmap be a mounted tmpfs or whatever, which we can more easily implement individual live-apply semantics.

[cgwalters]

Trying to play around with this, the ergonomics are definitely somewhat annoying because configmaps can't have / in keys. If one wanted to e.g. write drop a a CA certificate into /etc/pki/ca-trust/source/anchors and also a local registry mirror config in /etc/containers/registries.conf.d it'd require separate configmaps with separate mountpoints. Maybe we just accept that for now.

Edit: Or, we could accept a List object of configmaps.

[cgwalters]

Also, we should add support for kernel arguments via ConfigMap too...it'd be a really good match.

[cgwalters]

Dug into this a bit more, and actually we have all the bits we need for client-side support for configmaps-in-registry today, which is awesome.

The main gap is documenting support for uploading them (xref containers/buildah#5091 )

[fabiendupont]

@cgwalters, apart from being closer to existing Kubernetes objects, what ConfigMap/Secret bring compared to systemd-confext?

Systemd expects to find the config extensions in a specific folder, e.g. /run/confexts/. It is possible to ship config OCI images in the OS image and mount them under /run/confexts/ with ComposeFS via systemd.mount units before the systemd-confext.service unit.

[cgwalters]

This is a complex topic. First, I think it makes sense to enable confext use on top of bootc.

However the other case here is I think many will want support for "lifecycle binding" configs and the OS, i.e. bootc update intends to update both as a transactional unit (and integrated with rollbacks by default); confext is an independent thing today from OS updates by default (which is good, we also want something like that).

And yes, there is the Kubernetes orientation versus the DDI stuff.

[cgwalters]

The other thing that seems obvious to me to support is actually supporting per node state, i.e. associate configmaps with a specific node. This couldn't be used for pre-kubelet state (static IPs, hostname) but I can imagine it being useful for things like dynamic tuning.

That said though, there's definitely a coordination issue for any config changes that imply disruption (hence require drains etc.); so ultimately there'd need to be something MCO-like that is managing rollout I would say.

[fabiendupont]

For clarity, when you say a ConfigMap, what would it look like? Currently, it's a single file or list of files with their content base64-encoded. So, I see it as just a way to store the data, but it could as well be the files in an OCI image that is mounted as an overlay for /. The result would be the same. And outside of a Kubernetes API, the notion of ConfigMap doesn't exist, so I don't see how it applies practically to a standalone machine running Fedora.

I like the idea of the bootc pack command to assemble multiple layers into a final image. But it sounds almost like a Buildah feature, possibly containers/buildah#5091.

[cgwalters]

So, I see it as just a way to store the data, but it could as well be the files in an OCI image that is mounted as an overlay for /.

Yes, though a key difference here is: if you're shipping binaries, you need to care about multi-arch in the general case. For uploading generic configuration, you don't. See #22 (comment)

And outside of a Kubernetes API, the notion of ConfigMap doesn't exist, so I don't see how it applies practically to a standalone machine running Fedora.

https://docs.podman.io/en/v4.2/markdown/podman-play-kube.1.html exists and is usable without an API server. See also https://www.redhat.com/en/blog/running-containers-cars which argues for reusing container-ecosystem-adjacent tooling without an api server.

[cgwalters]

https://docs.podman.io/en/v4.2/markdown/podman-play-kube.1.html exists and is usable without an API server.

Also these things are connected, because it'd make sense to support attaching bootc configmaps which contain podman-play-kube definitions!

[jlebon]

@jmarrero the FROM scratch is more of a "fallback" path if we don't have OCI artifacts support. One big reason for this is that configuration by nature is almost always architecture independent, and having to do multi-arch builds for it would be annoying.

But, OTOH the FROM scratch approach does natively work as part of a multi-stage build to scrape in content.

So perhaps we should support both.

Yeah, I definitely see the argument for trying to keep this clean and using OCI artifacts, which seem like a better fit. That said, UX-wise having configs just be a regular container image too is very tempting. Building them is trivial in any container image build infra and it's more intuitive to write since the files live as files as they would on the target system rather than compiled into one big YAML file. So 👍 to supporting both.

Re. multi-arch, since bootc knows that it's just config files (given that you passed it to e.g. bootc config add), it doesn't have to care about trying to match arches. So maybe it can do something like "if it's manifest-listed, then try to match arch, otherwise just take the default".

[cgwalters]

and it's more intuitive to write since the files live as files as they would on the target system rather than compiled into one big YAML file.

Yes though we could easily add bootc config build|push

[alexlarsson]

https://docs.podman.io/en/v4.2/markdown/podman-play-kube.1.html exists and is usable without an API server.

Just for the record, quadlet supports .kube files for running play-kube things easily too.

[fabiendupont]

And outside of a Kubernetes API, the notion of ConfigMap doesn't exist, so I don't see how it applies practically to a standalone machine running Fedora.

https://docs.podman.io/en/v4.2/markdown/podman-play-kube.1.html exists and is usable without an API server. See also https://www.redhat.com/en/blog/running-containers-cars which argues for reusing container-ecosystem-adjacent tooling without an api server.

Do you plan to support other APIs?
I mean, Podman kube play supports Pod, Deployment, PersistentVolumeClaim, ConfigMap, Secret and DaemonSet. Which ones would you add to bootc?

Also, the Kubernetes doc says that ConfigMap are limited to 1MiB:

A ConfigMap is not designed to hold large chunks of data. The data stored in a ConfigMap cannot exceed 1 MiB. If you need to store settings that are larger than this limit, you may want to consider mounting a volume or use a separate database or file service.

[cgwalters]

Podman kube play supports Pod, Deployment, PersistentVolumeClaim, ConfigMap, Secret and DaemonSet. Which ones would you add to bootc?

I think bootc's role is just scoped to host management, so Pod/Deployment/DaemonSet are clearly out of scope, right? Those things should be handled by podman. Whether we do mapping around PersistentVolume for the host seems more open, but my instincts say that's just confusing and we should point administrators to systemd .mount units. ConfigMap and Secret to me are the most obvious things to support.

[cheesesashimi]

Files have a lot of metadata such as file mode, paths, owners, etc. Rather than encode all of this into an arbitrary JSON blob and stuff it inside a ConfigMap, why not use the metadata part of the ConfigMap to store these values as annotations?

Here's a rough idea of some annotations that could be used for this purpose:

Annotation Name:	Description:
bootc.coreos.io/baseDir	The base directory where each file in the ConfigMap will be created, e.g. /etc. This is required because the ConfigMap spec does not allow slashes in the name portion of the files it creates.
bootc.coreos.io/baseDirMode	The octal file mode (0755, 0644, etc.) that the deepest directory in the path should have. Given /etc/a/deeply/nested/base, this would affect the ./base portion of the directory mode.
bootc.coreos.io/fileMode	The octal file mode that each file in the ConfigMap should be created with (e.g., 0755, 0644, etc.). Additional file modes such as sticky bit, executable, et. al. should be supported. If multiple files within the same directory need to have different modes, more than one ConfigMap will be needed.
bootc.coreos.io/userOwner	The username or UID of the file's owner.
bootc.coreos.io/groupOwner	The group name or GID of the file's owner.

[cgwalters]

(From an organizational perspective note bootc is part of the containers/ GH org, not coreos/)

bootc.coreos.io/baseDir

Right, this one already exists in the WIP code...I think maybe what we should do is make a shared hackmd doc and refine it into a spec, then update the first comment here or so?

As far as the other metadata...maybe. So far, there hasn't been a use case for it in Kubernetes, right? I think specifically having executables in configmaps is probably something we should think of as an anti-pattern - we have container images for that. This reduces the need significantly for fileMode. The use cases for user/group owners also seem like they need very specific motivation.

[arewm]

To flesh this out a bit more, bootc upgrade would check for updates to these things too.

Would it make sense to enable only the configmap to be updated without updating the bootc image as well or would these actions/lifecycles be bound together somehow?

Would you be able to use an image manifest to store the config maps with the config descriptor could be used to describe how all of the layers should be applied to the file system?

These could then be customized per-architecture and distributed as an image index so a configuration file could be applied to the bootc installations regardless of the architecture.

[jlebon]

Here are some of the ways we've discussed so far in how configs could be packaged:

• configmap (YAML hosted somewhere or in an OCI artifact)
• FROM scratch container image
• systemd-confext (I guess... as an image file hosted somewhere or in an OCI artifact)

ISTM like those are completely orthogonal to how they're bound to the host (in the case of systemd-confext, it could be none at all, but... it could also be "managed" like the other options would be). Even in the managed option, there's e.g. configured in /etc (local state) or in /usr (closer to LBIs).

[jlebon]

I've mentioned this before OOB, but just to surface it here, from the CoreOS side we're very interested in this capability because there the default entrypoint is not to do a custom build, but the day-2 config management is still extremely useful. IOW, we want this so as to reduce the need for either having to reprovision (current Ignition model) or having to maintain a CI pipeline (bootc model).

[jlebon]

We had some in-person discussions related to this last week. Just wanted to jot down some of what was mentioned:

• Strongly related to remote config via configmap and secrets #22 (comment) and as mentioned in on-cluster builds enhancement openshift/enhancements#1515, this has important implications for on-cluster layering in OpenShift. Currently we're building a derived images containing MachineConfig content. This means that as soon as a user changes a MachineConfig, we have to trigger a new build. If we instead have a separate way to apply config changes, it means a lot less churn in how often we need to rebuild the full image.
• Following on the previous point, this might be out of scope for this issue but it'd be really nice if this just extended to /var as well. This would allow the MCO to get out of the business of writing files directly almost entirely without breaking backwards compatibility with what is currently possible with MachineConfigs. Obviously the implementation details would be different from the /etc case. E.g. we'd need to also think about reconciliation there, etc...