Add (and possibly require) ostree container commit

[cgwalters]

The current layered container code performs "ostree importing" client side - we accept arbitrary layer tarballs that could come from a Dockerfile or other container build system.

Let's consider adding a command line ostree container container that is required to be invoked
in the container at the end of a build process. For example in a Dockerfile:

FROM quay.io/fedora/coreos
ADD somefile.conf /etc/systemd/system
RUN curl ... && yum install .. && ostree container commit

The last step would effectively do most of the heavy lifting we're already doing client side:

• Move /etc -> /usr/etc (with backcompat link)
• Discard content outside of /usr - or better, error out
• Perform selinux labeling
• Compute ostree sha256

In this, I think the biggest benefit is that we can error out if people try to do things like ship content in /boot or /var at build time. But avoiding doing selinux labeling on the client is a not-small benefit too.

Note that in order for this to work, it must be invoked for each layer.

[cgwalters]

This issue also relates to ostreedev/ostree#2480 so the CLI can just be ostree instead of ostree-ext-cli.

[lucab]

Below some notes from an out-of-band chat that I forgot to write down earlier:

• this sounds like a subset of what rpm-ostree compose postprocess does. There is possibly some logic which could be shared between the two.
• rpm-ostree does automatic translation of /var into tmpfiles.d fragment too. But I'm unsure whether the logic here does assume systemd-tmpfiles (and other systemd mechanisms in general).
• depending on the starting image, it may not readily be in a runnable state. For example, FCOS images require some inflating steps (for processing tmpfiles.d, sysusers.d, and more). This in-container flow may need a symmetrical prepare/finalize logical scope to take care of that.

[cgwalters]

One thing that would likely ease implementation of this significantly is if we structure it as:

• First enhance the https://docs.rs/ostree-ext/latest/ostree_ext/tar/fn.import_tar.html API to support loading selinux policy from the running root i.e. / instead of requiring a base commit
• generate tarball stream from /usr in the running container using the https://docs.rs/tar/ API
• This would include things like transforming etc into usr/etc OR we could first mutate the running root
• pipe that tar stream to our own tar import API

And that may be it!

[cgwalters]

EDIT: actually what I wrote above is wrong because the ostree C code doesn't know about the tar format or how to write it.

We discussed a simpler option to start of just adding ostree container commit as a verb now and only having it do validation to start - for example, erroring out on content in /var that will be ignored.

(Also, a whole thread here around whether we should allow new toplevel directories outside of /usr or not - I think today we do)

Then later we can actually move all the commit logic mentioned above from per-machine (client side) to build time i.e. ostree container commit.

[cgwalters]

The above landed in #205

But let's leave this issue open to doing

Then later we can actually move all the commit logic mentioned above from per-machine (client side) to build time i.e. ostree container commit.

[srstsavage]

Just to check on where this stands, is it correct that derived Dockerfiles should use

RUN ostree container commit

as the final RUN command, and that it currently just does some sanity checking but may do more build time preprocessing in the future?

[cgwalters]

Hi, thanks for the ping here! I think we'd forgotten to follow up on this because it'd stalled a little bit on getting the new functionality shipped out, but that has happened now.

So I updated coreos/layering-examples#5 as one example.

But yes, please do start adding the command now! Does that make sense?

I'll see if we can update documentation in this repository too. I fixed the example Dockerfile here too - it can't just be RUN ostree container commit because that would normally create its own separate layer; instead it needs to be at the end of each RUN invocation. (Alternatively, in many cases where Dockerfile layers don't provide much value, one can just have a RUN build.sh that does everything)

[srstsavage]

Thanks! This does make sense. Two questions though:

it can't just be RUN ostree container commit because that would normally create its own separate layer; instead it needs to be at the end of each RUN invocation

If we don't mind the separate layer, is there any harm in just running ostree container commit as its own RUN at the end of the Dockerfile? To me that seems a bit cleaner/more maintainable. Is the main concern adding bloat in the layers once ostree container commit starts moving files around (e.g. /etc -> /usr/etc)?

I think the biggest benefit is that we can error out if people try to do things like ship content in /boot or /var at build time

Should ostree container commit be returning non-zero when it finds files in /var? Currently it reports them to stderr but returns 0, so Docker builds still succeed.

$ docker build --pull --no-cache - <<EOF
FROM quay.io/coreos-assembler/fcos:next-devel
RUN touch /var/bad && ostree container commit
EOF

Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM quay.io/coreos-assembler/fcos:next-devel
next-devel: Pulling from coreos-assembler/fcos
Digest: sha256:5b556d343a9d527f9f1e336e09ee0b75f52d1894cd585d17b31bcd7b3392503e
Status: Image is up to date for quay.io/coreos-assembler/fcos:next-devel
 ---> 3662acb8d28c
Step 2/2 : RUN touch /var/bad && ostree container commit
 ---> Running in b1e4f1829a37
Checking /var for files
Found file: "/var/bad"
Removing intermediate container b1e4f1829a37
 ---> 0f0408790ce6
Successfully built 0f0408790ce6

[cgwalters]

If we don't mind the separate layer, is there any harm in just running ostree container commit as its own RUN at the end of the Dockerfile?

That won't help unless the build is done with podman|docker build --squash (which is not the default, and also not configurable in every environment that accepts Dockerfile as input).

Remember, docker/OCI containers are just tarballs with JSON metadata. A layer is just a tarball.

So without squashing, we'd end up with a series of layers (tarballs) and then one final one which shuffles things around - and the client has to download all those tarballs - duplicating space.

On the client, we still need to handle the non-ostree-committed layers because we can't "know" that the files were changed in a later layer.

In other words, having RUN ostree container commit as a separate layer would currently be an anti-optimization i.e. pessimization.

[cgwalters]

Tangentially related to this in lieu of better docs elsewhere: It will not work to invoke e.g. useradd foo in a Dockerfile today, because we entirely drop all content in /var when importing the image. This relates to https://ostreedev.github.io/ostree/adapting-existing/#system-layout

ostree makes a strong guarantee it will not affect user data on upgrades, and /home is user data.

[cgwalters]

Should ostree container commit be returning non-zero when it finds files in /var? Currently it reports them to stderr but returns 0, so Docker builds still succeed.

Also sorry...I missed this comment before, and it's now fixed in #364 along with #367

I think where we are now with ostree container commit is sufficient to close this.

However: it seems very likely that we will want to drain some logic that lives in rpm-ostree today for things like generating tmpfiles.d and sysusers.d snippets from data.