Rootless docker & podman build fails on Linux with /usr/bin/ld: cannot open output file /project/x: Permission denied

[parawanderer]

Describe the bug

The issue

I was trying to build a project using my default docker setup on my Ubuntu 22.04.3 LTS laptop and got this issue during the docker-based native build step. The issue happens with both GraalVM and Mandrel via this configuration, as well as using both podman and docker.

I understand it is related to the rootless mode not assigning the permissions properly to be able to read/write to /project/x inside the container.

It appears to be close to or the same issue as in #33188 and #10637

Some debugging I've done

If I set it to use podman using the property <quarkus.native.container-runtime>podman</quarkus.native.container-runtime> and copy the command it is running from full output log and add --userns=keep-id after the --user 1000:1000 parameter, the command succeeds.

I am not sure if any similar solution exists for rootless docker.

I have also looked at the current logic for determining when to add --userns=keep-id and am confused as to why it is not being included in my parameters list when I configure it to use podman:

quarkus/core/deployment/src/main/java/io/quarkus/deployment/pkg/steps/NativeImageBuildLocalContainerRunner.java

Line 31 in f974941

containerRuntimeArgs.add("--userns=keep-id");

I cloned this project at the latest main and ran the tests in NativeImageBuildContainerRunnerTest and believe that it is not properly detecting me to be using rootless podman, seemingly because of the quarkus-local-container-runtime system property being set here:

quarkus/core/runtime/src/main/java/io/quarkus/runtime/util/ContainerRuntimeUtil.java

Line 132 in f974941

final ContainerRuntime containerRuntime = ContainerRuntime.of(runtime);

Which then immediately returns it here without determining if it is "rootless" or not (below):

quarkus/core/runtime/src/main/java/io/quarkus/runtime/util/ContainerRuntimeUtil.java

Line 43 in f974941

ContainerRuntime containerRuntime = loadContainerRuntimeFromSystemProperty();

ContainerRuntime.of() seems like it will match the first enum value with "podman" as the name, which I believe will always be the one with rootless set to false here:

quarkus/core/runtime/src/main/java/io/quarkus/runtime/util/ContainerRuntimeUtil.java

Line 218 in f974941

PODMAN("podman", false),

Therefore it never adds the --userns.

As I've found a workaround and it is not extremely clear to me how to properly contribute to this project, I will not be pursuing an MR to try and fix the above issue.

Expected behavior

The build should work as it does with non-rootless mode docker.

Actual behavior

I get a failure inside the container due to being unable to access path /project/x:

Full output may be found back here

/usr/bin/ld: cannot open output file /project/my-proj-1.0-SNAPSHOT-runner: Permission denied
collect2: error: ld returned 1 exit status
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.image.NativeImageViaCC.handleLinkerFailure(NativeImageViaCC.java:203)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.image.NativeImageViaCC.runLinkerCommand(NativeImageViaCC.java:151)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.image.NativeImageViaCC.write(NativeImageViaCC.java:117)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:718)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:535)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:403)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:580)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:128)

How to Reproduce?

1. Run Ubuntu as the host OS with docker desktop and podman desktop. Use the default configuration for both, which appears to be "rootless" mode.
2. Add quarkus native compilation as a build step in maven
3. Invoke this build step, in my case mvn clean install -U
4. The final step of the build process for a module then invokes docker (the step name was quarkus:3.1.1.Final:build) and fails with the above error

It is unclear to me if the quarkus-local-container-runtime System Property is being set anywhere as I am not setting it in my project nor in my ~/.m2/settings.xml file. But this being set seems like it might be a prerequisite too.

Output of uname -a or ver

Linux parawanderer-Latitude-7430 6.2.0-32-generic #32~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Aug 18 10:40:13 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

Output of java -version

openjdk version "20.0.2" 2023-07-18 OpenJDK Runtime Environment Homebrew (build 20.0.2) OpenJDK 64-Bit Server VM Homebrew (build 20.0.2, mixed mode, sharing)

GraalVM version (if different from Java)

No response

Quarkus version or git rev

3.1.1.Final

Build tool (ie. output of mvnw --version or gradlew --version)

Apache Maven 3.9.1 (2e178502fcdbffc201671fb2537d0cb4b4cc58f8) Maven home: /home/linuxbrew/.linuxbrew/Cellar/maven/3.9.1/libexec Java version: 17.0.6, vendor: Azul Systems, Inc., runtime: /home/parawanderer/zulu17.40.19-ca-jdk17.0.6-linux_x64 Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "6.2.0-32-generic", arch: "amd64", family: "unix"

Additional information

A workaround for me has been to use docker instead of podman, and change my docker context (docker context list) to default, which seems to be non-rootless mode:

docker context use default

After this, the build succeeded.

Before doing this change, it was set to using desktop-linux for me, which in the docker context list output was reported as the following:

NAME                TYPE                DESCRIPTION                               DOCKER ENDPOINT                                    KUBERNETES ENDPOINT   ORCHESTRATOR
default             moby                Current DOCKER_HOST based configuration   unix:///var/run/docker.sock                                              
desktop-linux *     moby                                                          unix:///home/parawanderer/.docker/desktop/docker.sock

Which I figured to be "rootless" mode, whereas default seemed to be "non-rootless".

[quarkus-bot]

/cc @Karm (mandrel), @galderz (mandrel), @geoand (kubernetes), @iocanel (kubernetes), @zakkak (mandrel)

[geoand]

Have you tried with newer versions of Quarkus?

[geoand]

Closing as we never got feedback