ERROR: IOError: stat: permission denied (EACCES) on udocker #34918
Comments
|
What happens if you start Julia with the flag For example: (inside the container) ./julia-1.3.1/bin/julia --startup-file=no |
|
|
Alright let’s try this: (inside the container) ./julia-1.3.1/bin/julia --startup-file=no --history-file=no |
|
It's a bit odd for |
|
No With |
|
First, try this: (inside the container) export JULIA_DEPOT_PATH="$HOME/.julia:"
./julia-1.3.1/bin/julia --startup-file=no --history-file=noIf that does not work, then try this instead: (inside the container) export JULIA_DEPOT_PATH="$HOME/.julia"
./julia-1.3.1/bin/julia --startup-file=no --history-file=no |
|
I have a few additional questions:
|
and both lead to Permissions of and finally: |
|
Can you run |
|
|
In the Julia REPL, try the following: julia> rm("/root/.julia"; force = true, recursive = true)
julia> mkpath("/root/.julia")
julia> open("/root/.julia/foo.txt", "w") do io
println(io, "hello world")
end
julia> println(read("/root/.julia/foo.txt", String))
julia> println(realpath("/root/.julia/foo.txt")) |
|
Also:
Oh, and also: julia> Base.stat("/")
julia> Base.stat("/root")
julia> Base.stat("/tmp")Also, is there any chance that your host system is running SELinux? |
There is no .julia dir, so I can't delete it no When I manually create I always start with a SELinux…I don't know. Tested so far on vanilla Ubuntu 19.10 and 20.04 hosts. |
|
Update: |
You can run that command in Julia whether or not the file/directory exists. In Julia, if you run the
You can run the following commands in getenforcesestatus |
|
This seems like a bug in one of the following:
|
|
What happens if you go into a brand-new container, enter a Julia REPL, and run this: julia> Base.stat("/root/.julia") |
|
Also, on one of the hosts that does have this error, can you download and extract Julia on the host (not inside udocker, not inside a container), open a Julia REPL, and try to reproduce this error? That will help us figure out of the problem is with your host machine or with udocker. |
|
Inside a brand new container: On the host (all machines in reach), I do not get any errors. Everything works as expected. Have there been substantial changes between julia 1.0.5 and the following versions with respect to directory creation/stat/security/rights management? |
|
What about this: (inside a brand new container) julia> run(`stat /root/.julia`)I expect this to throw an error. The question is: will it throw an error because of |
I'm not sure. Is Julia 1.0.5 the most recent version that works for you? Do any of the Julia 1.1.x versions work? In particular, if you can confirm that Julia 1.0.5 works for you, but Julia 1.1.0 gives this error, then we can try to look at the diff between those two versions. |
|
Here's another thing to try. Inside a fresh container: julia> Base.stat("/root/.foo")
julia> Base.stat("/root/foo") |
|
|
I tried different versions:
|
|
Can you try julia-1.3.0 also? |
|
@Keno does the fact that gives "no such file or directory" (the correct answer), but gives "permission denied (EACCES)" suggest that this is a Julia bug or a libuv bug (rather than a problem with the system)? |
|
|
So, if you are able to build Julia from source inside the container, the best option will be to |
|
The script for the git bisect can probably be as simple as: make
rm -rf /root/.julia
./julia -e 'Base.stat("/root/.julia") |
|
Pass that script to |
|
To add the reference, as I suspected there is no documented difference between stat and statx with respect to eaccess errors: http://man7.org/linux/man-pages/man2/statx.2.html. @dr-br could you show the output of |
|
mount inside container: |
|
That's inside the container? Looks more like a host system. I wonder if it's picking up the outside world instead. |
|
To recall this:
The problems seem to be the combination Ubuntu host + Udocker @Keno: Yes, udocker is not really what you expect, if you know docker or podman. |
In Julia 1.2.0 and older, the version of libuv bundled with Julia did not use Starting with Julia 1.3.0, the version of libuv bundled with Julia uses |
|
@dr-br So CentOS host and Ubuntu container is fine. But Ubuntu host and Ubuntu container has the problem. I wonder: could you try Ubuntu host and CentOS container, and CentOS host and CentOS container? I am curious: does this bug happen whenever the host Linux distro is the same as the container Linux distro? Or does it only happen for the very specific Ubuntu-Ubuntu combination? |
|
So I just built 1.5.0-DEV on our RHEL 7.7 Xeon Gold 6230 cluster inside an Ubuntu 18.04 udocker container. Works as expected, no errors. @DilumAluthge: I already tried Ubuntu host and CentOS container, it gave the same errors. |
|
Might be a kernel bug in the Ubuntu host kernel |
|
@Keno: I would not dare to blame the Ubuntu host kernel. I mean, udocker does so many magic tricks. I would rather think of udocker not being well enough tested on "consumer-OSses". |
|
Meh, we find kernel bugs about once or twice a month around these parts ;), but yes at this point this might be something to take up with the udocker developers. |
|
It would be nice to have a MWE. Maybe a simple C program that makes both of the syscalls ( |
|
@dr-br I notice that you have this in your workflow: export PROOT_NO_SECCOMP=1I wonder if that is relevant. |
|
I also wonder if this issue is relevant: https://bugs.launchpad.net/ubuntu/+source/docker.io/+bug/1755250 |
Very likely. On the RHEL machines this is not necessary, only on the Ubuntu hosts. |
|
What is the latest version of Ubuntu that you have access to? |
|
It seems to be statx inside container. The Ubuntu versions I ran all the above tests are 19.10 and 20.04. They behave the same. |
|
@Keno Is there any chance that in the Julia-specific fork of libuv, we could stop using Alternatively, we could ask upstream libuv to stop using |
|
This is not unique to udocker. For example, this issue: docker/for-linux#208
|
|
Funny thing: I don't even get the statx example compiled on the RHEL host, as there is currently a 3.10 kernel running ;) |
|
I think that JuliaLang/libuv#7 will fix this. |
After a discussion with my colleague: Is it possible, that libuv already has a fallback, if statx ist not available? How else could julia run on a RHEL 7 host with 3.10 kernel? |
If your kernel is old and does not have As you can see in the code, the fallback only applies when the errno returned by If |
|
The errno returned by seccomp is user defined. It not being enosys or something sensible is a bug in udocker or one of its dependencies. |
I’ve corrected my answer. |
@dr-br Can you post an issue on the udocker repo (https://github.com/indigo-dc/udocker) and include the code for your example programs? And cc me in the issue? Hopefully that will help us get things moving on the udocker end. |
|
Since this turned out not to be a julia issue, I'm gonna go ahead and close this. Discussion can continue here of course. |
Intro+Relevance
udocker is a basic user tool to execute simple docker containers in user space without requiring root privileges.
It is the only means to deploy containerized jupyter+X on our super computers.
podman, docker and singularity can not be used (subuid/guid issues…) on the server nodes.
Problem
Current Julia versions (including nightly build) fail to run within a udocker-container. The container OSses testet are ubuntu (18.04) and centos:latest.
Older Julia versions like 1.0.5 and older perfectly work.
Tests with podman or docker are successful for all versions of Julia.
When I execute julia inside a udocker container I get the following error message (the same under ubuntu and centos):
Steps to reproduce
Install udocker
Start ubuntu container
Download and run Julia
Within the ubuntu container run:
The text was updated successfully, but these errors were encountered: