overlayfs is not working on Debian host

[douggutaby]

Description
I cannot run pip upgrade in an alpine image on a debian host with overlayfs.

Steps to reproduce the issue:
I use the following Dockerfile:

FROM python:3.10.10-slim-bullseye
USER root

# some certs because we use jfrog remote repos
COPY root.pem intermediate.pem /usr/local/share/ca-certificates/
RUN for f in /usr/local/share/ca-certificates/*.pem; do cp -- "$f"  "${f%.pem}.crt"; done
RUN update-ca-certificates

# pip.config for our jfrog remote repo with these two lines
# [global]
# index-url = https://USER:PASS@JFROG_URL....
COPY pip.conf /etc/

RUN cat /usr/local/share/ca-certificates/intermediate.pem /usr/local/share/ca-certificates/root.pem > /tmp/allcert.pem
ENV REQUESTS_CA_BUNDLE=/tmp/allcert.pem

RUN python3 -m ensurepip
RUN pip3 install --verbose --upgrade pip

Describe the results you received:
If I run buildah bud -t testimage . with non-root user, I got the following:

...
STEP 10/10: RUN pip3 install --upgrade pip
Looking in indexes: https://****/artifactory/api/pypi/pip-remote/simple
Requirement already satisfied: pip in /usr/local/lib/python3.10/site-packages (22.3.1)
Collecting pip
Downloading https://****/artifactory/api/pypi/pip-remote/packages/packages/08/e3/57d4c24a050aa0bcca46b2920bff40847db79535dc78141eb83581a52eb8/pip-23.1.2-py3-none-any.whl (2.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 20.9 MB/s eta 0:00:00
IInstalling collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 22.3.1
    Uninstalling pip-22.3.1:
      Removing file or directory /usr/local/bin/pip
      Removing file or directory /usr/local/bin/pip3
      Removing file or directory /usr/local/bin/pip3.10
      Removing file or directory /usr/local/lib/python3.10/site-packages/pip-22.3.1.dist-info/
ERROR: Could not install packages due to an OSError.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/shutil.py", line 816, in move
    os.rename(src, real_dst)
OSError: [Errno 18] Invalid cross-device link: '/usr/local/lib/python3.10/site-packages/pip/' -> '/usr/local/lib/python3.10/site-packages/~ip'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/pip/_internal/commands/install.py", line 494, in run
    installed = install_given_reqs(
...

It uses overlayfs, but if I set --storage-driver overlay --storage-opt mount_program=/usr/bin/fuse-overlayfs, the result is the same.

Describe the results you expected:
Buf I do the same with root, or with non-root, but with STORAGE_DRIVER=vfs, or run pip upgrade with non-root in the Dockerfile it is working:

...
STEP 10/10: RUN pip3 install --upgrade pip
Looking in indexes: https://****/artifactory/api/pypi/pip-remote/simple
Requirement already satisfied: pip in /usr/local/lib/python3.10/site-packages (22.3.1)
Collecting pip
Downloading https://****/artifactory/api/pypi/pip-remote/packages/packages/08/e3/57d4c24a050aa0bcca46b2920bff40847db79535dc78141eb83581a52eb8/pip-23.1.2-py3-none-any.whl (2.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 35.1 MB/s eta 0:00:00
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 22.3.1
Uninstalling pip-22.3.1:
Successfully uninstalled pip-22.3.1
Successfully installed pip-23.1.2
...

Output of rpm -q buildah or apt list buildah:
I'm using the latest buildah, but it is the same if I install buildah 1.19 with apt.
buildah/stable 1.19.6+dfsg1-1+b6 amd64

Output of buildah version:

Version: 1.31.0-dev
Go Version: go1.20.4
Image Spec: 1.1.0-rc.3
Runtime Spec: 1.1.0-rc.2
CNI Spec: 1.0.0
libcni Version: v1.1.2
image Version: 5.25.1-dev
Git Commit: c3b7dcb8482bc226a8a3accf9e113238f91cfa93
Built: Thu Jun 1 14:52:57 2023
OS/Arch: linux/amd64
BuildPlatform: linux/amd64

Output of cat /etc/*release:

PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Output of uname -a:
Linux itdevvmtest 5.10.0-23-amd64 #1 SMP Debian 5.10.179-1 (2023-05-12) x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:
We don't have these config files.

cat: /etc/containers/storage.conf: No such file or directory
cat: /home/user/.config/containers/storage.conf: No such file or directory

[flouthoc]

Hi, Could you share more detail like if you are running buildah inside a container ? output of buildah info ?

[douggutaby]

Hi,

I don't run buildah inside a container. It is a new Debian VM, and if I install buildah with apt (1.19) or I build from source (1.31), the result is the same.
I added --verbose to pip upgrade to my first comment and here is the buildah info:

{
    "host": {
        "CgroupVersion": "v2",
        "Distribution": {
            "distribution": "debian",
            "version": "11"
        },
        "MemFree": 346910720,
        "MemTotal": 4109250560,
        "OCIRuntime": "runc",
        "SwapFree": 2996563968,
        "SwapTotal": 2998923264,
        "arch": "amd64",
        "cpus": 2,
        "hostname": "itdevvmtest",
        "kernel": "5.10.0-23-amd64",
        "os": "linux",
        "rootless": true,
        "uptime": "20h 43m 28.59s (Approximately 0.83 days)",
        "variant": ""
    },
    "store": {
        "ContainerStore": {
            "number": 1
        },
        "GraphDriverName": "overlay",
        "GraphOptions": null,
        "GraphRoot": "/home/user/.local/share/containers/storage",
        "GraphStatus": {
            "Backing Filesystem": "extfs",
            "Native Overlay Diff": "false",
            "Supports d_type": "true",
            "Using metacopy": "false"
        },
        "ImageStore": {
            "number": 2
        },
        "RunRoot": "/var/tmp/containers-user-1002/containers"
    }
}

If I run the same with root, this line is different:
"Native Overlay Diff": "true",

[rhatdan]

How old of a kernel are you running with?

Could you change the /etc/containers/storage.conf to specify overlay as the storage driver?

[douggutaby]

apt upgrade, update run a few days ago. Output of uname -a:
Linux itdevvmtest 5.10.0-23-amd64 #1 SMP Debian 5.10.179-1 (2023-05-12) x86_64 GNU/Linux
As I know the rootless buildah doesn't use the /etc/containers/storage.conf just the config in the HOME.If I put the storage settings in etc it doesn't work for non root user.
I used the --storage-driver overlay or STORAGE_DRIVER variable. Aren't they the same as the storage.conf? The result was the same.

[douggutaby]

I checked on Debian 12 and the build is working with overlay. So the problem is only on Debian 11.

[rhatdan]

Most likely rootless overlay does not work on Debian 11.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.