CoreOS layering: useradd during package installation happens in the container, not target system

[vrutkovs]

Host system details

# rpm-ostree --version
rpm-ostree:
 Version: '2022.2'
 Git: 1e62f2bb60bc39b443dac0c00d40124ef002bb73
 Features:
  - rust
  - compose
  - fedora-integration

Expected vs actual behavior

CoreOS layering in containers incorrectly executes post-install scripts in some packages:

# cat Dockerfile   
FROM quay.io/coreos-assembler/fcos:testing-devel
RUN rpm-ostree install openvswitch && id openvswitch
# podman build -t custom-fcos . -f Dockerfile               
STEP 1/2: FROM quay.io/coreos-assembler/fcos:testing-devel
STEP 2/2: RUN rpm-ostree install openvswitch && id openvswitch
Enabled rpm-md repositories: fedora-modular fedora updates fedora-cisco-openh264 updates-modular updates-archive
...
Installing: openvswitch-2.15.0-7.fc35.x86_64 (fedora)
uid=985(openvswitch) gid=976(openvswitch) groups=976(openvswitch),975(hugetlbfs)
COMMIT custom-fcos

openvswitch apparently runs useradd during post-install script, so it runs in container instead of target system.

[cgwalters]

Yes, this is going to be a really common problem. This relates to ostreedev/ostree-rs-ext#159 in general, but also ye olde #49 in particular.

So specifically though we need to support RUN ostree container finalize going through the code that lives in rpm-ostree today around /usr/lib/passwd in the same way it does for RPMs; this way we would support direct RUN useradd --system invocations too.

At a practical level it may be simplest to have something like /usr/lib/ostree/finalize.d/ hooks and stick rpm-ostree in there instead of trying to drain down all the passwd handling code.

[cgwalters]

By far the best thing though is for things to stop running useradd at all, and instead rely on e.g. systemd DynamicUser=yes. That seems like a pretty good fit for openvswitch.

[cgwalters]

One thing that's slowly happening is packages are adding systemd-sysusers fragments, which will basically make this work again because the user will be re-added on firstboot of the final system.

This issue also ultimately intersects strongly with ostreedev/ostree#2868 because that's the best end state in a layering world - the source of truth should be the container image (and possibly attached configmaps).