Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

failed to create rootless cni resolv.conf: open .../resolv.conf: no such file or directory #10930

Closed
edsantiago opened this issue Jul 14, 2021 · 1 comment · Fixed by #10939
Closed
Assignees
Labels
flakes Flakes from Continuous Integration In Progress This issue is actively being worked by the assignee, please do not work on this at this time. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. rootless

Comments

@edsantiago
Copy link
Member

I think this is now our number-one flake:

Running: podman [options] run --name con1 --network IntTesta6a1514e70245da0670fc583887a7057165a3da069ad872452ca38bbcf957c44 quay.io/libpod/alpine:latest nslookup con1
Error: failed to create rootless cni resolv.conf: open /run/user/13502/rootless-cni/resolv.conf: no such file or directory

Once this happens, the system is forevermore broken, but further attempts give a different error:

Error: failed to mount netns directory for rootless cni: no such file or directory

It's a hard one to track down because once it fails, many other tests fail. I will try to follow up here after our f2f.

Podman network [It] podman network prune

Podman run networking [It] podman run check dnsname plugin

@edsantiago edsantiago added flakes Flakes from Continuous Integration rootless labels Jul 14, 2021
@Luap99
Copy link
Member

Luap99 commented Jul 14, 2021

I know what is going on. It is a race between the cleanup and setup for the rootless cni namespace. I think I know how to fix this.

@Luap99 Luap99 self-assigned this Jul 14, 2021
@Luap99 Luap99 added the In Progress This issue is actively being worked by the assignee, please do not work on this at this time. label Jul 15, 2021
Luap99 added a commit to Luap99/libpod that referenced this issue Jul 15, 2021
There was an race condition when calling `GetRootlessCNINetNs()`. It
created the rootless cni directory before it got locked. Therefore
another process could have called cleanup and removed this directory
before it was used resulting in errors. The lockfile got moved into the
XDG_RUNTIME_DIR directory to prevent a panic when the parent dir was
removed by cleanup.

Fixes containers#10930
Fixes containers#10922

To make this even more robust `GetRootlessCNINetNs()` will now return
locked. This guarantees that we can run `Do()` after `GetRootlessCNINetNs()`
before another process could have called `Cleanup()` in between.

[NO TESTS NEEDED] CI is flaking, hopefully this will fix it.

Signed-off-by: Paul Holzinger <[email protected]>
Luap99 added a commit to Luap99/libpod that referenced this issue Jul 15, 2021
There was an race condition when calling `GetRootlessCNINetNs()`. It
created the rootless cni directory before it got locked. Therefore
another process could have called cleanup and removed this directory
before it was used resulting in errors. The lockfile got moved into the
XDG_RUNTIME_DIR directory to prevent a panic when the parent dir was
removed by cleanup.

Fixes containers#10930
Fixes containers#10922

To make this even more robust `GetRootlessCNINetNs()` will now return
locked. This guarantees that we can run `Do()` after `GetRootlessCNINetNs()`
before another process could have called `Cleanup()` in between.

[NO TESTS NEEDED] CI is flaking, hopefully this will fix it.

Signed-off-by: Paul Holzinger <[email protected]>
@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 21, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 21, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
flakes Flakes from Continuous Integration In Progress This issue is actively being worked by the assignee, please do not work on this at this time. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. rootless
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants