Simplify and fix os.MkdirAll() usage #162
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
TL;DR: check for IsExist(err) after a failed MkdirAll() is both
redundant and wrong -- so two reasons to remove it.
Quoting MkdirAll documentation:
> MkdirAll creates a directory named path, along with any necessary
> parents, and returns nil, or else returns an error. If path
> is already a directory, MkdirAll does nothing and returns nil.
This means two things:
1. If a directory to be created already exists, no error is
returned.
2. If the error returned is IsExist (EEXIST), it means there exists
a non-directory with the same name as MkdirAll need to use for
directory. Example: we want to MkdirAll("a/b"), but file "a"
(or "a/b") already exists, so MkdirAll fails.
The above is a theory, based on quoted documentation and my UNIX
knowledge.
3. In practice, though, current MkdirAll implementation [1] returns
ENOTDIR in most of cases described in #2, with the exception when
there is a race between MkdirAll and someone else creating the
last component of MkdirAll argument as a file. In this very case
MkdirAll() will indeed return EEXIST.
Because of #1, IsExist check after MkdirAll is not needed.
Because of #2 and #3, ignoring IsExist error is just plain wrong,
as directory we require is not created. It's cleaner to report
the error now.
Note this error is all over the tree, I guess due to copy-paste,
or trying to follow the same usage pattern as for Mkdir(),
or some not quite correct examples on the Internet.
[1] https://github.com/golang/go/blob/f9ed2f75/src/os/path.go
Signed-off-by: Kir Kolyshkin <[email protected]>
|
Here's an example |
|
@kolyshkin You're right, I actually don't see how it can return |
|
@LK4D4 It actually can return EEXIST in a case of the following race (which I was lazy to explain in details, but since you asked for it...). First, MkdirAll calls stat() and gets ENOENT, so it goes on to check and create all the parents. At this time, someone else creates a file under the same name as the last MkdirAll component. Finally, MkdirAll calls Mkdir to create that last component, Mkdir returns EEXIST which MkdirAll passes to the caller. |
|
@kolyshkin Totally, thanks for explanation! |
LK4D4
added a commit
that referenced
this pull request
Jul 30, 2015
Simplify and fix os.MkdirAll() usage
stefanberger
pushed a commit
to stefanberger/runc
that referenced
this pull request
Sep 8, 2017
*.md: markdown formatting
kolyshkin
added a commit
to kolyshkin/storage
that referenced
this pull request
May 16, 2020
This subtle bug keeps lurking in because error checking for `Mkdir()` and `MkdirAll()` is slightly different wrt `EEXIST`/`IsExist`: - for `Mkdir()`, `IsExist` error should (usually) be ignored (unless you want to make sure directory was not there before) as it means "the destination directory was already there"; - for `MkdirAll()`, `IsExist` error should NEVER be ignored. This commit removes ignoring the IsExist error, as it should not be ignored. For more details, a quote from my opencontainers/runc#162 (July 2015): -quote- TL;DR: check for IsExist(err) after a failed MkdirAll() is both redundant and wrong -- so two reasons to remove it. Quoting MkdirAll documentation: MkdirAll creates a directory named path, along with any necessary parents, and returns nil, or else returns an error. If path is already a directory, MkdirAll does nothing and returns nil. This means two things: If a directory to be created already exists, no error is returned. If the error returned is IsExist (EEXIST), it means there exists a non-directory with the same name as MkdirAll need to use for directory. Example: we want to MkdirAll("a/b"), but file "a" (or "a/b") already exists, so MkdirAll fails. The above is a theory, based on quoted documentation and my UNIX knowledge. In practice, though, current MkdirAll implementation [1] returns ENOTDIR in most of cases described in containers#2, with the exception when there is a race between MkdirAll and someone else creating the last component of MkdirAll argument as a file. In this very case MkdirAll() will indeed return EEXIST. Because of containers#1, IsExist check after MkdirAll is not needed. Because of containers#2 and containers#3, ignoring IsExist error is just plain wrong, as directory we require is not created. It's cleaner to report the error now. Note this error is all over the tree, I guess due to copy-paste, or trying to follow the same usage pattern as for Mkdir(), or some not quite correct examples on the Internet. [1] https://github.com/golang/go/blob/f9ed2f75/src/os/path.go -end-quote- Signed-off-by: Kir Kolyshkin <[email protected]>
This was referenced May 16, 2020
kolyshkin
added a commit
to kolyshkin/storage
that referenced
this pull request
May 18, 2020
This subtle bug keeps lurking in because error checking for `Mkdir()` and `MkdirAll()` is slightly different wrt `EEXIST`/`IsExist`: - for `Mkdir()`, `IsExist` error should (usually) be ignored (unless you want to make sure directory was not there before) as it means "the destination directory was already there"; - for `MkdirAll()`, `IsExist` error should NEVER be ignored. This commit removes ignoring the IsExist error, as it should not be ignored. For more details, a quote from my opencontainers/runc#162 (July 2015): -quote- TL;DR: check for IsExist(err) after a failed MkdirAll() is both redundant and wrong -- so two reasons to remove it. Quoting MkdirAll documentation: MkdirAll creates a directory named path, along with any necessary parents, and returns nil, or else returns an error. If path is already a directory, MkdirAll does nothing and returns nil. This means two things: If a directory to be created already exists, no error is returned. If the error returned is IsExist (EEXIST), it means there exists a non-directory with the same name as MkdirAll need to use for directory. Example: we want to MkdirAll("a/b"), but file "a" (or "a/b") already exists, so MkdirAll fails. The above is a theory, based on quoted documentation and my UNIX knowledge. In practice, though, current MkdirAll implementation [1] returns ENOTDIR in most of cases described in containers#2, with the exception when there is a race between MkdirAll and someone else creating the last component of MkdirAll argument as a file. In this very case MkdirAll() will indeed return EEXIST. Because of containers#1, IsExist check after MkdirAll is not needed. Because of containers#2 and containers#3, ignoring IsExist error is just plain wrong, as directory we require is not created. It's cleaner to report the error now. Note this error is all over the tree, I guess due to copy-paste, or trying to follow the same usage pattern as for Mkdir(), or some not quite correct examples on the Internet. [1] https://github.com/golang/go/blob/f9ed2f75/src/os/path.go -end-quote- Signed-off-by: Kir Kolyshkin <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Please review; I find it moderately serious as missing error from MkdirAll() can lead to some other errors at a later stage, possibly hard to track down to the cause.