Buildah failed to build image with additionalimage options #2359

bulanovk · 2020-05-14T15:43:41Z

We faced an issue when try to reproduce images cache approach from article https://developers.redhat.com/blog/2019/08/14/best-practices-for-running-buildah-in-a-container/?intcmp=701f20000012ngPAAQ&extIdCarryOver=true&sc_cid=7013a000002DTxFAAW

The main idea it to use additionalimage options, as a result on Fedora 32 base os with
podman 1.9.1 we got following error:

STEP 1: FROM centos:7
STEP 2: RUN echo "1" >/1.txt
STEP 3: COMMIT x
ERRO error unmounting /var/lib/shared/overlay/b05580fca2f9aabb2d8fa975b29146c9147c8418e559f197c54a4fac04babb95/merged: invalid argument 
error committing container for step {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] Command:run Args:[echo "1" >/1.txt] Flags:[] Attrs:map[] Message:RUN echo "1" >/1.txt Original:RUN echo "1" >/1.txt}: error copying layers and metadata for container "35379374a9d5e3e124d3c31800d1b41ebe16cf49a3b8f1ef3c416ad3e285aad5": Error initializing source containers-storage:centos-working-container: error extracting layer "580cfaeeb9478ef0b4bbbc67558ac75b342f1c1fc4722147f8fac6570bc563c3": error creating overlay mount to /var/lib/shared/overlay/b05580fca2f9aabb2d8fa975b29146c9147c8418e559f197c54a4fac04babb95/merged: using mount program /usr/bin/fuse-overlayfs: fuse-overlayfs: cannot open workdir: No such file or directory
: exit status 1

Script to reproduce

rm -rf /tmp/c2 /tmp/c1
mkdir -p /tmp/c1 /tmp/c2 /tmp/w
cd /tmp/w
cat << EOF > Dockerfile
FROM centos:7
RUN echo "1" >/1.txt
EOF
podman run -it --rm --privileged=true -v /tmp/c2:/var/lib/containers:z quay.io/buildah/stable buildah pull centos:7
podman run -it --rm --privileged=true -v /tmp/c2/storage:/var/lib/shared:ro -v /tmp/c1:/var/lib/containers:z -v $(pwd):/wrk:z quay.io/buildah/stable sh -c "cd /wrk && buildah bud -t x ."

The text was updated successfully, but these errors were encountered:

rhatdan · 2020-05-14T16:59:05Z

Are you runnint podman as root or rootless?

bulanovk · 2020-05-14T17:31:11Z

podman was running as root

rhatdan · 2020-05-14T19:14:13Z

I am getting the same failure. @giuseppe Any ideas?

rhatdan · 2020-05-14T19:24:50Z

Seems to be an issue with buildah bud. If I run each command separately everything works.

bulanovk · 2020-05-15T05:08:56Z

I also try to build this example in a script way

rm -rf /tmp/c2 /tmp/c1
mkdir -p /tmp/c1 /tmp/c2 /tmp/w
cd /tmp/w
cat << EOF > build.sh
  cd /wrk && \
  newcontainer=\$(buildah from centos:7) && \
  buildah run \$newcontainer sh -c 'echo "1" >/1.txt' && \
  buildah --log-level debug commit \$newcontainer x 
EOF
podman run -it --rm --privileged=true -v /tmp/c2:/var/lib/containers:z quay.io/buildah/stable buildah pull centos:7
podman run -it --rm --privileged=true -v /tmp/c2/storage:/var/lib/shared:ro -v /tmp/c1:/var/lib/containers:z -v $(pwd):/wrk:z quay.io/buildah/stable sh -c '
  cd /wrk && \
  sh ./build.sh'

but got follwoing error

ERRO error unmounting /var/lib/shared/overlay/b05580fca2f9aabb2d8fa975b29146c9147c8418e559f197c54a4fac04babb95/merged: invalid argument 
error committing container "centos-working-container-1" to "x": error copying layers and metadata for container "ac11dcd4039af54ae4fb8bdd7d1f8137c5bbfb580e45af1f7f1ea2abcd97a60b": Error initializing source containers-storage:centos-working-container-1: error extracting layer "5313273facfff6a40213fdceb87c414052e7eacbcf25e0026055e0e839ba73b7": error creating overlay mount to /var/lib/shared/overlay/b05580fca2f9aabb2d8fa975b29146c9147c8418e559f197c54a4fac04babb95/merged: using mount program /usr/bin/fuse-overlayfs: fuse-overlayfs: cannot open workdir: No such file or directory
: exit status 1

Debug log from commit command

DEBU [graphdriver] trying provided driver "overlay" 
DEBU overlay: imagestore=/var/lib/shared          
DEBU overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU backingFs=tmpfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU Loading registries configuration "/etc/containers/registries.conf" 
DEBU parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.imagestore=/var/lib/shared,overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]localhost/x:latest" 
DEBU committing image with reference "containers-storage:[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.imagestore=/var/lib/shared,overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]localhost/x:latest" is allowed by policy 
DEBU parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.imagestore=/var/lib/shared,overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]@8114e7c1868bd9b15188df8c34953b151b81f90a73f16bc00aaec227bd500cc9" 
DEBU base image "8114e7c1868bd9b15188df8c34953b151b81f90a73f16bc00aaec227bd500cc9" is already present in local storage, no need to copy its layers 
DEBU layer list: ["b05580fca2f9aabb2d8fa975b29146c9147c8418e559f197c54a4fac04babb95" "5313273facfff6a40213fdceb87c414052e7eacbcf25e0026055e0e839ba73b7"] 
DEBU using "/var/tmp/buildah662391787" to hold temporary data 
DEBU overlay: mount_data=nodev,fsync=0,lowerdir=/var/lib/shared/overlay/l/UCEFSJ7JN7HRA7WDFDGY62JGIT,upperdir=/var/lib/containers/storage/overlay/5313273facfff6a40213fdceb87c414052e7eacbcf25e0026055e0e839ba73b7/diff,workdir=/var/lib/containers/storage/overlay/5313273facfff6a40213fdceb87c414052e7eacbcf25e0026055e0e839ba73b7/work 
DEBU overlay: mount_data=ro,lowerdir=/var/lib/shared/overlay/b05580fca2f9aabb2d8fa975b29146c9147c8418e559f197c54a4fac04babb95/empty,upperdir=/var/lib/shared/overlay/b05580fca2f9aabb2d8fa975b29146c9147c8418e559f197c54a4fac04babb95/diff,workdir=/var/lib/shared/overlay/b05580fca2f9aabb2d8fa975b29146c9147c8418e559f197c54a4fac04babb95/work 
ERRO error unmounting /var/lib/shared/overlay/b05580fca2f9aabb2d8fa975b29146c9147c8418e559f197c54a4fac04babb95/merged: invalid argument 
error committing container "centos-working-container-1" to "x": error copying layers and metadata for container "ac11dcd4039af54ae4fb8bdd7d1f8137c5bbfb580e45af1f7f1ea2abcd97a60b": Error initializing source containers-storage:centos-working-container-1: error extracting layer "5313273facfff6a40213fdceb87c414052e7eacbcf25e0026055e0e839ba73b7": error creating overlay mount to /var/lib/shared/overlay/b05580fca2f9aabb2d8fa975b29146c9147c8418e559f197c54a4fac04babb95/merged: using mount program /usr/bin/fuse-overlayfs: fuse-overlayfs: cannot open workdir: No such file or directory
: exit status 1

bulanovk · 2020-05-15T05:42:44Z

And one more interesting thing. when fuse-overlay was turned off, buildah was able to perform build, but fails to push to registry

Error

error pushing image "docker.cloud-lab.cf:5001/kb/bh:b01" to "docker://docker.cloud-lab.cf:5001/kb/bh:b01": error copying layers and metadata from "containers-storage:[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.imagestore=/var/lib/shared,overlay.mountopt=nodev]docker.cloud-lab.cf:5001/kb/bh:b01" to "docker://docker.cloud-lab.cf:5001/kb/bh:b01": Error reading blob sha256:b05580fca2f9aabb2d8fa975b29146c9147c8418e559f197c54a4fac04babb95: not allowed to update mount locations for layers at "/var/run/containers/storage/overlay-layers/mountpoints.json": called a write method on a read-only store

script

rm -rf /tmp/c2 /tmp/c1
mkdir -p /tmp/c1 /tmp/c2 /tmp/w
cd /tmp/w
cat << EOF > build.sh
  cd /wrk && \
  sed -i -e 's|^mount_program|#mount_program|g' /etc/containers/storage.conf
  sed -i 's+^mountopt.*\$+mountopt = "nodev"+g' /etc/containers/storage.conf
  echo "LOG:newcontainer"  
  newcontainer=\$(buildah from centos:7) && \
  echo "LOG:buildah run "
  buildah run \$newcontainer sh -c 'echo "1" >/1.txt' && \
  echo "LOG:buildah commit"
  buildah --log-level debug commit \$newcontainer nexus:5001/kb/bh:b01 
  echo "LOG:buildah login"
  buildah login -u admin -p admin123 nexus:5001
  echo "LOG:buildah push"	
  buildah --log-level debug push nexus:5001/kb/bh:b01
EOF
podman run -it --rm --privileged=true -v /tmp/c2:/var/lib/containers:z quay.io/buildah/stable buildah pull centos:7
podman run -it --rm --privileged=true -v /tmp/c2/storage:/var/lib/shared:ro -v /tmp/c1:/var/lib/containers:z -v $(pwd):/wrk:z quay.io/buildah/stable sh -c '
  cd /wrk && \
  sh ./build.sh'

full log: https://gist.github.com/bulanovk/953b2afdef1dc9d3199a6ef6c4d1dd07

giuseppe · 2020-05-15T07:38:45Z

fuse-overlayfs needs write access to the shared repository to create the work directory and operate there when mounting the image. It must be mounted using -v /tmp/c2/storage:/var/lib/shared:rw

@rhatdan is it possible to mount the image read only?

bulanovk · 2020-05-15T07:52:41Z

fuse-overlayfs needs write access to the shared repository to create the work directory and operate there when mounting the image. It must be mounted using -v /tmp/c2/storage:/var/lib/shared:rw

Yes it helps, but it still fails in push

Script:

rm -rf /tmp/c2 /tmp/c1
mkdir -p /tmp/c1 /tmp/c2 /tmp/w
cd /tmp/w
cat << EOF > Dockerfile
FROM centos:7
RUN echo "1" >/1.txt
EOF
podman run -it --rm --privileged=true -v /tmp/c2:/var/lib/containers:z quay.io/buildah/stable buildah pull centos:7
podman run -it --rm --privileged=true -v /tmp/c2/storage:/var/lib/shared:z -v /tmp/c1:/var/lib/containers:z -v $(pwd):/wrk:z quay.io/buildah/stable sh -c "cd /wrk && buildah bud -t nexus:5001/kb/bh:b01 . && buildah login -u admin -p admin123 nexus:5001/kb/bh:b01 && buildah --log-level error push nexus:5001/kb/bh:b01"

error pushing image "nexus:5001/kb/bh:b01" to "docker://nexus:5001/kb/bh:b01": error copying layers and metadata from "containers-storage:[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.imagestore=/var/lib/shared,overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]nexus:5001/kb/bh:b01" to "docker://nexus:5001/kb/bh:b01": Error reading blob sha256:b05580fca2f9aabb2d8fa975b29146c9147c8418e559f197c54a4fac04babb95: not allowed to update mount locations for layers at "/var/run/containers/storage/overlay-layers/mountpoints.json": called a write method on a read-only store

full log: https://gist.github.com/bulanovk/953b2afdef1dc9d3199a6ef6c4d1dd07#file-bud_build-log

rhatdan · 2020-05-15T15:58:26Z

@giuseppe I think that is a bug in fuse-overlay. The shared container images should be readonly. I think this would work with regular overlay. The shared images are for the lower level, why is fuse-overlay writing to this directory? /var/lib/containers should container the upper/writable layer.

giuseppe · 2020-05-15T19:13:09Z

@giuseppe I think that is a bug in fuse-overlay. The shared container images should be readonly. I think this would work with regular overlay. The shared images are for the lower level, why is fuse-overlay writing to this directory? /var/lib/containers should container the upper/writable layer.

I am seeing fuse-overlayfs used with an upperdir and a workdir specified:

/usr/bin/fuse-overlayfs -o lowerdir=/var/lib/shared/overlay/b05580fca2f9aabb2d8fa975b29146c9147c8418e559f197c54a4fac04babb95/empty,upperdir=/var/lib/shared/overlay/b05580fca2f9aabb2d8fa975b29146c9147c8418e559f197c54a4fac04babb95/diff,workdir=/var/lib/shared/overlay/b05580fca2f9aabb2d8fa975b29146c9147c8418e559f197c54a4fac04babb95/work /var/lib/shared/overlay/b05580fca2f9aabb2d8fa975b29146c9147c8418e559f197c54a4fac04babb95/merged

rhatdan · 2020-05-15T20:00:42Z

Any idea why, I don't see this happen when I do a buildah mount.

rhatdan · 2020-05-15T20:05:43Z

I can not get this to happen outside of the container.

rhatdan · 2020-05-15T20:09:52Z

If I remove fuse-overlay and the mountoptions from containers/storage.conf, it works.

rhatdan · 2020-05-15T20:36:11Z

I think some how fuse-overlay is getting confused on the upper directory and mount directory.

bulanovk · 2020-05-16T06:48:45Z

If I remove fuse-overlay and the mountoptions from containers/storage.conf, it works.

With overlay it will fails to push image to remote registry, if additionalimage not empty and FROM image are r/o

if "ro" is specified when mounting overlay, configure overlay without any upper layer and workdir so the file system doesn't attempt to write any file. Closes: containers/buildah#2359 Signed-off-by: Giuseppe Scrivano <[email protected]>

giuseppe · 2020-05-16T10:05:06Z

this PR fixes the issue for me when attempting to create a workdir on a ro storage: containers/storage#628

I've tried with fuse-overlayfs, but I expect it to fix the root case as well.

bulanovk · 2020-05-19T04:52:31Z

Guys, is there a chance to get fix in buildah release?
@giuseppe did you try to push result image to registry with your fix?

giuseppe · 2020-05-19T08:21:13Z

@giuseppe did you try to push result image to registry with your fix?

the push seems still broken, I'll take a look

giuseppe · 2020-05-19T09:22:49Z

we can fix it for overlay and vfs with: containers/storage#631

Unfortunately it is not a generic fix as other backends still require to mount the layer.

gzm55 · 2020-10-15T17:16:36Z

@giuseppe in rootless container, buildah 1.16.4 with fuse-overlayfs driver still fail to commit image, the error log:

error copying layers and metadata for container "d626c35e8d5ef6624bebd73c697547e4cb7a48febd506e258e8cb017b4a24ef2":
Error initializing source containers-storage:alpine-working-container:
error extracting layer "6dce341435fb76a4ff72077e070f1ed3c276c016d317a60a2ef1372de8c93004":
chown /storage/overlay/50644c29ef5a27c9a40c393a73ece2479de78325cae7d762ef3cdc19bf42dd0a/diff: read-only file system

In root mode, building works as excepted. /storage is a mounted ro dir, the owner is root.

~ $ mount | grep /storage
grpcfuse on /storage type fuse.grpcfuse (ro,relatime,user_id=0,group_id=0,allow_other,max_read=1048576)
~ $ ls -alh /storage
total 8K
drwxr-xr-x   11 testuser testuser     352 Oct 15 14:42 .
drwxr-xr-x    1 root     root        4.0K Oct 15 17:05 ..
drwxr-xr-x    3 root     root          96 Oct 15 14:42 cache
drwxr-xr-x    2 root     root          64 Oct 15 14:42 mounts
drwxr-xr-x    4 root     root         128 Oct 15 14:42 overlay
drwxr-xr-x    3 root     root          96 Oct 15 14:42 overlay-containers

giuseppe · 2020-10-26T07:41:36Z

was the shared storage created with an older version of buildah or podman?

In any case, I don't think it can work if it is owned by root. The rootless user doesn't have enough privileges to read all the files in the images.

The shared storage must be owned by the same rootless user and every file in the storage owned by a UID/GID available in the rootless user namespace. Otherwise you need to make sure every file has at least mode 0755 when owned by a different user.

gzm55 · 2020-10-26T07:54:52Z

was the shared storage created with an older version of buildah or podman?

In any case, I don't think it can work if it is owned by root. The rootless user doesn't have enough privileges to read all the files in the images.

The shared storage must be owned by the same rootless user and every file in the storage owned by a UID/GID available in the rootless user namespace. Otherwise you need to make sure every file has at least mode 0755 when owned by a different user.

The problem occurs on the latest podman/buildah. it seems that the grpcfuse fs will report error if a moved file does not have a writable permission.

## on MacOS, mount a not user folder into container, the grpcfuse will shift the uid of the mounted folder to 0.
touch /1.txt
chmod 444 /1.txt
mv /1.txt /mnt ## report error, return non-zero, but the file is moved correctly with correct permission

Since some folder do not have write permission, such as /proc in alpine, the previous problem when dealing with the permission will block the un-tar commands, then the buildah load fails.

mfblair · 2021-09-24T22:00:42Z

Hello All,

I seem to have hit an issue possibly related to the one in this thread on RHEL 8.3, and upgrading to 8.4 did not resolve it. I'm using rootless containers and additionalimage options as described in this article .

As a rootless user, when trying to build a new intermediate image in a multi-stage a Dockerfile, I'm trying to pull from an image in the Read Only shared area /var/lib/mycontainers. I get this error:

Error initializing source containers-storage:vcdevhub-working-container-1: error extracting layer "925c7db0fc455b5d4d7ac03683a0b0d0b71d73f63aeb93b33026aadb3fda1c72": chown /var/lib/mycontainers/overlay/f62732f7ab4af0fd05b173fadc39a8659085a1c21d43dd652ad1e63e698b8f9c/diff: operation not permitted

It seems as though it's trying to update (chown) in the shared area? I get the same error whether I use podman or buildah. (From what I've read, podman actually uses buildah code for building).

Note. The error does not happen when I try the same operation as root.

Here is some version info:
$ cat /etc/redhat-release
Red Hat Enterprise Linux release 8.4 (Ootpa)

$ uname -a
Linux localhost.localdomain 4.18.0-305.19.1.el8_4.x86_64 #1 SMP Tue Sep 7 07:07:31 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux

podman version 3.2.3
buildah version 1.21.4 (image-spec 1.0.1-dev, runtime-spec 1.0.2-dev)

Does anyone know whether the fix for the issue in this thread should be implemented in RHEL 8.4?

Thanks,
Blair

rhatdan assigned giuseppe May 14, 2020

bulanovk changed the title ~~Buildah failed to build image with additional images~~ Buildah failed to build image with additionalimage options May 15, 2020

giuseppe mentioned this issue May 16, 2020

overlay: honor ro option containers/storage#628

Merged

vrothberg closed this as completed in containers/storage#628 May 17, 2020

giuseppe reopened this May 19, 2020

giuseppe mentioned this issue May 19, 2020

overlay: implement DiffGetter containers/storage#631

Merged

rhatdan closed this as completed in rhatdan/storage@cb48094 May 20, 2020

github-actions bot added the locked - please file new issue/PR label Sep 3, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 3, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Buildah failed to build image with additionalimage options #2359

Buildah failed to build image with additionalimage options #2359

bulanovk commented May 14, 2020

rhatdan commented May 14, 2020

bulanovk commented May 14, 2020

rhatdan commented May 14, 2020

rhatdan commented May 14, 2020

bulanovk commented May 15, 2020 •

edited

Loading

bulanovk commented May 15, 2020 •

edited

Loading

giuseppe commented May 15, 2020 •

edited

Loading

bulanovk commented May 15, 2020

rhatdan commented May 15, 2020

giuseppe commented May 15, 2020

rhatdan commented May 15, 2020

rhatdan commented May 15, 2020

rhatdan commented May 15, 2020 •

edited

Loading

rhatdan commented May 15, 2020

bulanovk commented May 16, 2020

giuseppe commented May 16, 2020 •

edited

Loading

bulanovk commented May 19, 2020

giuseppe commented May 19, 2020

giuseppe commented May 19, 2020

gzm55 commented Oct 15, 2020

giuseppe commented Oct 26, 2020

gzm55 commented Oct 26, 2020 •

edited

Loading

mfblair commented Sep 24, 2021

Buildah failed to build image with additionalimage options #2359

Buildah failed to build image with additionalimage options #2359

Comments

bulanovk commented May 14, 2020

rhatdan commented May 14, 2020

bulanovk commented May 14, 2020

rhatdan commented May 14, 2020

rhatdan commented May 14, 2020

bulanovk commented May 15, 2020 • edited Loading

bulanovk commented May 15, 2020 • edited Loading

giuseppe commented May 15, 2020 • edited Loading

bulanovk commented May 15, 2020

rhatdan commented May 15, 2020

giuseppe commented May 15, 2020

rhatdan commented May 15, 2020

rhatdan commented May 15, 2020

rhatdan commented May 15, 2020 • edited Loading

rhatdan commented May 15, 2020

bulanovk commented May 16, 2020

giuseppe commented May 16, 2020 • edited Loading

bulanovk commented May 19, 2020

giuseppe commented May 19, 2020

giuseppe commented May 19, 2020

gzm55 commented Oct 15, 2020

giuseppe commented Oct 26, 2020

gzm55 commented Oct 26, 2020 • edited Loading

mfblair commented Sep 24, 2021

bulanovk commented May 15, 2020 •

edited

Loading

bulanovk commented May 15, 2020 •

edited

Loading

giuseppe commented May 15, 2020 •

edited

Loading

rhatdan commented May 15, 2020 •

edited

Loading

giuseppe commented May 16, 2020 •

edited

Loading

gzm55 commented Oct 26, 2020 •

edited

Loading