FATA[0000] Error writing blob: open /var/lib/containers/storage/overlay/0f49addb6e505b120dc9caa86a06503ec3664a1adb3cbbd9ba720a861d16f851/merged/dev/null: no such file or directory

[thomasmckay]

Description

# skopeo copy containers-storage:1008a4a8be1c dir:buildahtest-tag1
Getting image source signatures
Copying blob sha256:0f49addb6e505b120dc9caa86a06503ec3664a1adb3cbbd9ba720a861d16f851
 2.00 KB / 452.97 MB [>-----------------------------------------------------] 0s
FATA[0000] Error writing blob: open /var/lib/containers/storage/overlay/0f49addb6e505b120dc9caa86a06503ec3664a1adb3cbbd9ba720a861d16f851/merged/dev/null: no such file or directory

Steps to reproduce the issue:

#!/bin/bash
IMAGE=devtable/anotheralpine:buildah3
BUILDAH_FORMAT=docker
BUILDAH_LAYERS=true
scratch_container=$(buildah from scratch)
scratch_mount=$(buildah mount $scratch_container)
yum install --installroot $scratch_mount --releasever 7 --setopt install_weak_deps=false --setopt tsflags=nodocs -y bash coreutils-single java-1.8.0-openjdk-headless
buildah unmount $scratch_container
buildah commit $scratch_container quay.example.com:5000/$IMAGE
buildah rm $scratch_container
podman push --creds devtable:password --tls-verify=false quay.example.com:5000/$IMAGE

[root@quay ~]# podman images
REPOSITORY                                     TAG        IMAGE ID       CREATED          SIZE
<none>                                         <none>     53db958b591a   25 minutes ago   475MB
quay.example.com:5000/devtable/anotheralpine   buildah3   1008a4a8be1c   25 minutes ago   475MB
quay.io/thomasmckay/buildahtest                tag1       1008a4a8be1c   25 minutes ago   475MB
quay.example.com:5000/devtable/anotheralpine   buildah2   172abfab5de8   37 minutes ago   475MB
[root@quay ~]# skopeo copy containers-storage:1008a4a8be1c dir:buildahtest-tag1
Getting image source signatures
Copying blob sha256:0f49addb6e505b120dc9caa86a06503ec3664a1adb3cbbd9ba720a861d16f851
 2.00 KB / 452.97 MB [>-----------------------------------------------------] 0s
FATA[0000] Error writing blob: open /var/lib/containers/storage/overlay/0f49addb6e505b120dc9caa86a06503ec3664a1adb3cbbd9ba720a861d16f851/merged/dev/null: no such file or directory

Describe the results you received:

Describe the results you expected:

Output of rpm -q buildah or apt list buildah:

buildah-1.2-2.gitbe87762.el7.x86_64

Output of buildah version:

Version:         1.2
Go Version:      go1.9.4
Image Spec:      1.0.0
Runtime Spec:    1.0.0
CNI Spec:        0.3.1
libcni Version:  v0.6.0
Git Commit:      be87762
Built:           Tue Aug 21 14:38:24 2018
OS/Arch:         linux/amd64

Output of podman version if reporting a podman build issue:

Version:       0.9.2
Go Version:    go1.9.4
OS/Arch:       linux/amd64

Output of skopeo --version

# skopeo --version
skopeo version 0.1.31-dev

[rhatdan]

@mtrmac PTAL

[mtrmac]

These open /var/lib/containers/storage/overlay/…/merged/…: no such file or directory storage corruptions keep popping up with an increasing frequency. Has anyone tracked the root cause down?

Hoping @nalind has an idea.

[zvonkok]

I have a similar issue

podman push quay.io/zvonkok/node-feature-discovery:v0.3.0-10-g86947fc-dirty
Getting image source signatures
Copying blob sha256:237472299760d6726d376385edd9e79c310fe91d794bc9870d038417d448c2d5
 1024 B / 55.73 MB [>-------------------------------------------------------] 0s
Error copying image to the remote destination: Error writing blob: Patch https://quay.io/v2/zvonkok/node-feature-discovery/blobs/uploads/5bdcd221-cbca-4007-9b0b-5c037c400b8d: open /var/lib/containers/storage/overlay/237472299760d6726d376385edd9e79c310fe91d794bc9870d038417d448c2d5/merged/bin/bash: no such file or directory

Tested with podman-0.9.2 and podman-0.10.2-dev. Build with buildah-1.2 and with podman. Both times I get this error.

[rhatdan]

We have not been hearing about these recently, are they still an issue with podman 1.11 and buildah 1.15

[rhatdan]

No response, Closing as we believe this is fixed.

[Fodoj]

Happened to me today as well:

Getting image source signatures
Copying blob d69483a6face done
Copying blob 011d008dd19f done
Error: Error copying image to the remote destination: Error writing blob: Patch https://registry-1.docker.io/v2/mkdevme/app/blobs/uploads/8385126b-e49e-45e8-8bfc-32dc01fb997c?_state=VDhAOwAlXV5rboXisT2jJXKPhWxvVDZyRqpjTxcTz6p7Ik5hbWUiOiJta2Rldm1lL2FwcCIsIlVVSUQiOiI4Mzg1MTI2Yi1lNDllLTQ1ZTgtOGJmYy0zMmRjMDFmYjk5N2MiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTktMDQtMjNUMTg6NTg6MDkuODY5NjAwODk3WiJ9: open /home/fodoj/.local/share/containers/storage/overlay/d69483a6face4499acb974449d1303591fcbb5cdce5420f36f8a6607bda11854/merged/anaconda-post.log: no such file or directory

[azatsarynnyy]

Hello! I've got this issue with Buildah 1.11.2. From the comments, I see that it may be related to the environment/configuration.
Could someone share how you fixed it?

[TomSweeneyRedHat]

I don't have a fix in hand, would love to hear from someone in the community who might. I will reopen this though. @nalind @mtrmac FYI.

[ibotty]

Maybe that might help in narrowing it down. This happened on a debian system when multiple processes pulled the same image. When only one process pulled it, everything worked alright.

[rhatdan]

@ibotty @Fodoj Still seeing this issue?

[Fodoj]

@rhatdan hard for me to remember how I reproduced it initially, but I would say its fixed in latest releases.

[ibotty]

I can try to trigger it again, hopefully friday. If not the week after next week. I worked around the problem.

[rhatdan]

Ok reopen if you retrigger.