rpm-ostree install fails for scriptlet with skopeo copy ... containers-storage: due to bwrap limitations

[mtalexan]

Describe the bug

Related to this old issue: #2675

We're including oci-dirs of containers in some custom RPMs, and want to be able to populate a containers-storage in /usr/share/containers/storage as part of the RPM install. We attempt to us skopeo with the overlay driver and fuse-overlayfs mount program during a %post scriptlet, but something in the current bwrap limitations causes it to fail with an overlayfs error.

---

To run skopeo ... containers-storage:... in the %post scriptlet, we've learned it's necessary to manually re-add root user to /etc/sub{u,g}id files because those files aren't mounted in from the existing file system during the scriptlet execution.

Additionally, the default runroot is not a writeable location in the bwrap environment and has to be moved somewhere writeable.

Reproduction steps

On a development system:

1. Setup the system for building local rpms (rpmbuild or mock)
2. Save a container image to an oci dir

skopeo copy docker://example.com/my_image:version oci:oci_dir:example.com/my_image:version

3. Tar up the oci dir and including it in the ~/rpmbuild/SOURCES folder

tar -ca -f ~/rpmbuild/SOURCES/oci_dir.tar.gz oci_dir

4. In an RPM specfile, list the source tarball as a source, unpack it, and install the contents into /usr/share/containers/%{name}/images

Source1: oci_dir.tar.gz

...

%prep
...
tar -xf ${SOURCE1}

...

%install
...
mkdir -p $RPM_BUILD_ROOT/%{_datadir}/containers/%{name}/images
cp -a oci_dir/*  $RPM_BUILD_ROOT/%{_datadir}/containers/%{name}/images

...

%files
...
%{_datadir}/containers/%{name}

5. In the RPM specfile, include a %post scriptlet that sets up and runs skopeo to copy the imaages from the installed oci_dir into a containers-storage at /usr/share/containers/storage.

Requires(post): jq skopeo fuse-overlayfs

...

%post
# This scriptlet runs in a bwrap environment, which shares some isolation properties with a container.
# Most notably:
#  - The current fs is an overlayfs
#  - /usr/etc and /etc are not available from the real system
#
# We want to use skopeo/podman to copy images into a different root that can later be used as an additionalImageStore with
# an 'overlay' driver.  This requires some setup of this ephemeral environment to allow the minimum necessary for skopeo/podman
# to do image copying.
#
# 1. Enable subids for the current (root) user
# 2. Setup a writeable (ephemeral) run_root
# 3. Use the fuse-overlayfs mount driver instead of the default overlayfs driver (so we can do overlay on overlay, as per podman/skopeo-in-container documentation: https://github.com/containers/image_build/blob/main/podman/Containerfile#L79-L83)
# 4. Temporarily set the graph_root to our destination

# Set subuid and subgid in our ephemeral /etc copy. Actual values aren't important to match anything, they're only used here and don't affect the results.
echo "root:10000:65535" >> /etc/subuid
echo "root:10000:65535" >> /etc/subgid

# Create a run_root folder we can write to from here. Make it somewhere that's thrown away after the scriptlet is done
temp_run_root=/tmp/containers
mkdir -p ${temp_run_root}

# sets driver=overlay, graphroot={_datadir}/containers/storage, runroot=${temp_run_root}, and storage.options.overlay.mount_program=/usr/bin/fuse-overlayfs.
# It's not possible to set storage.options.overlay.mountopt this way, the options being set have to be comma separated, but the value for that option is also comma separated and they conflict.
#
# Alternatively, these values can be put in /etc/containers/storage.conf and they are demonstrated to have the identical effect.
skopeo --debug copy "oci:%{_datadir}/containers/%{name}/images:example.com/my_image:version" "containers-storage:[overlay@%{_datadir}/containers/storage+${temp_run_root}:mount_program=/usr/bin/fuse-overlayfs]example.com/my_image:version" || exit 1

6. Build the RPM
7. Copy the RPM to a running Fedora Atomic system
8. On the Fedora Atomic system: Install the RPM

...
rpm-ostree install ./my-rpm-1.0.0-0.fc40.x86_64.rpm

Expected behavior

The images are copied into the /usr/share/containers/storage/overlay folder in the overlay format.

Actual behavior

Install fails, and journalctl lists this as the reason during the skopeo --debug copy ... command.

filesystem type 0x65735546 reported for /usr/share/containers/storage is not supported with 'overlay': backing file system is unsupported for this graph driver

System details

rpm-ostree:
Version: '2022.19'
Git: f86a68c39eed6171c51cf8f2cab6c51aba94c05d
Features:

• rust
• compose
• container
• fedora-integration

Additional information

Using podman and skopeo within containers has a documented and working use case: https://github.com/containers/image_build/blob/main/podman/Containerfile#L79-L83
This is always an execution on top of an overlayfs, so the claim that this isn't possible that's made in buildah is incorrect: containers/buildah#4810
The major change is to use the mount_program = "/usr/bin/fuse-overlayfs" for the overlay driver options, which allows mounting of overlays on top of an overlaysfs (according to that documentation). (Weirdly, that's exactly what was reported as being used in the buildah Issue, but they still closed it as "not possible" even though it demonstrably is).

I'm unclear what permissions might be missing to enable skopeo to actually use the fuse-overlayfs (which it presumably isn't/can't?), but that's the heart of the bug report.

---

For the particular use case I've also tried using the vfs driver and that runs into other issues with being unable to extract certain gzipped layers, but that's out of scope of this bug report.

[mtalexan]

After some debugging, the issue appears to be that the bwrap scriptlet environment doesn't include /dev/fuse. So the fuse-overlayfs can't be used properly.