fix MkdirAll usage #2361
Closed
fix MkdirAll usage #2361
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This is causing a test failure? |
|
Looks like if the destination directory is a linkfile, then we would expect it to also succeed, at least in this case. |
|
@kolyshkin How do you want to fix this? |
|
is this still being worked on? |
Yes. Sorry I've neglected this one. Looking... |
|
OK, this happens because MkdirAll is executed on the host, but the symlink is absolute and only makes sense in a chroot. I can think of two ways to fix this, but I don't like either. It is probably best to ignore the error and (in case it's not actually a directory) rely on failing during IOW we have something like this: Obviously, the symlink is correct from within the container's chroot, but since MkdirAll() is called from the host context (without chroot), it sees the symlink as invalid and reports the EEXIST error which we see here. I have added a commit which ignores the error from MkdirAll only in this particular case, but frankly I am not sure if it brings much value. The only value is we fail earlier rather than later (when actually trying to So, second patch is optional and can be removed; please let me know what you think. |
cben
referenced
this pull request
Jun 16, 2020
Add a pull request template. Modeled after CRI-O (Thanks @saschagrunert!) and Dockers. Signed-off-by: TomSweeneyRedHat <[email protected]>
|
@kolyshkin can you try rebasing this please? We'd a fix in #2427 that should help at least some if not all of the test issues. |
This subtle bug keeps lurking in because error checking for `Mkdir()` and `MkdirAll()` is slightly different wrt `EEXIST`/`IsExist`: - for `Mkdir()`, `IsExist` error should (usually) be ignored (unless you want to make sure directory was not there before) as it means "the destination directory was already there"; - for `MkdirAll()`, `IsExist` error should NEVER be ignored. This commit removes ignoring the IsExist error, as it should not be ignored. [v2: skip patching (*Builder).Run] For more details, a quote from opencontainers/runc PR containers#162: -quote- TL;DR: check for IsExist(err) after a failed MkdirAll() is both redundant and wrong -- so two reasons to remove it. Quoting MkdirAll documentation: > MkdirAll creates a directory named path, along with any necessary > parents, and returns nil, or else returns an error. If path > is already a directory, MkdirAll does nothing and returns nil. This means two things: 1. If a directory to be created already exists, no error is returned. 2. If the error returned is IsExist (EEXIST), it means there exists a non-directory with the same name as MkdirAll need to use for directory. Example: we want to MkdirAll("a/b"), but file "a" (or "a/b") already exists, so MkdirAll fails. The above is a theory, based on quoted documentation and my UNIX knowledge. 3. In practice, though, current MkdirAll implementation [1] returns ENOTDIR in most of cases described in containers#2, with the exception when there is a race between MkdirAll and someone else creating the last component of MkdirAll argument as a file. In this very case MkdirAll() will indeed return EEXIST. Because of containers#1, IsExist check after MkdirAll is not needed. Because of containers#2 and containers#3, ignoring IsExist error is just plain wrong, as directory we require is not created. It's cleaner to report the error now. Note this error is all over the tree, I guess due to copy-paste, or trying to follow the same usage pattern as for Mkdir(), or some not quite correct examples on the Internet. > [1] https://github.com/golang/go/blob/f9ed2f75/src/os/path.go -end-quote- Signed-off-by: Kir Kolyshkin <[email protected]>
It is not entirely correct to always ignore EEXIST here. It should only be ignored in one special case: when a working directory already exists, and is an absolute symlink to another directory under container root. MkdirAll reports an error because the symlink is broken in the host context (without chroot). Signed-off-by: Kir Kolyshkin <[email protected]>
done; thanks for looking into it @TomSweeneyRedHat |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: kolyshkin The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
@kolyshkin Would you like to rebase this so we can get it in? |
openshift-merge-robot
added a commit
that referenced
this pull request
Nov 5, 2020
Attempt to test `fix MkdirAll usage #2361`
|
I completed this PR here: #2735 |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This subtle bug keeps lurking in because error checking for
Mkdir()and
MkdirAll()is slightly different wrtEEXIST/IsExist:for
Mkdir(),IsExisterror should (usually) be ignored(unless you want to make sure directory was not there before)
as it means "the destination directory was already there";
for
MkdirAll(),IsExisterror should NEVER be ignored.This commit removes ignoring the IsExist error, as it should not
be ignored.
For more details, a quote from opencontainers/runc#162:
-quote-
TL;DR: check for IsExist(err) after a failed MkdirAll() is both
redundant and wrong -- so two reasons to remove it.
Quoting MkdirAll documentation:
This means two things:
If a directory to be created already exists, no error is
returned.
If the error returned is IsExist (EEXIST), it means there exists
a non-directory with the same name as MkdirAll need to use for
directory. Example: we want to MkdirAll("a/b"), but file "a"
(or "a/b") already exists, so MkdirAll fails.
The above is a theory, based on quoted documentation and my UNIX
knowledge.
ENOTDIR in most of cases described in Needs to support creating multiple layers #2, with the exception when
there is a race between MkdirAll and someone else creating the
last component of MkdirAll argument as a file. In this very case
MkdirAll() will indeed return EEXIST.
Because of #1, IsExist check after MkdirAll is not needed.
Because of #2 and #3, ignoring IsExist error is just plain wrong,
as directory we require is not created. It's cleaner to report
the error now.
Note this error is all over the tree, I guess due to copy-paste,
or trying to follow the same usage pattern as for Mkdir(),
or some not quite correct examples on the Internet.
-end-quote-
Signed-off-by: Kir Kolyshkin [email protected]
What type of PR is this?
What this PR does / why we need it:
How to verify it
Which issue(s) this PR fixes:
Special notes for your reviewer:
Does this PR introduce a user-facing change?