Make FsBlobContainer Listing Resilient to Concurrent Modifications #49142
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
If we list out files in a folder via the lazily computed directory stream, we have to deal with concurrent deletes when reading the file attributes since we don't have a lock on the directory in any way. Closes elastic#37581
original-brownbear
added
>bug
:Distributed/Snapshot/Restore
|
Pinging @elastic/es-distributed (:Distributed/Snapshot/Restore) |
|
Jenkins run elasticsearch-ci/1 |
ywelsch
suggested changes
Nov 15, 2019
| final BasicFileAttributes attrs; | ||
| try { | ||
| attrs = Files.readAttributes(file, BasicFileAttributes.class); | ||
| } catch (FileNotFoundException e) { |
There was a problem hiding this comment.
This might need to be NoSuchFileException | FileNotFoundException.
| try { | ||
| attrs = Files.readAttributes(file, BasicFileAttributes.class); | ||
| } catch (FileNotFoundException e) { | ||
| continue; |
There was a problem hiding this comment.
can you add a comment here saying why it's ok to ignore?
| attrs = Files.readAttributes(file, BasicFileAttributes.class); | ||
| } catch (FileNotFoundException e) { | ||
| continue; | ||
| } | ||
| if (attrs.isRegularFile()) { |
There was a problem hiding this comment.
perhaps easier as the two suggestions above is to use Files.isRegularFile(file) here which takes care of all this.
There was a problem hiding this comment.
This won't help us I think? We still want the file size two lines below where we'd have to make another call to the FS that we'd have to guard against "file not found".
| attrs = Files.readAttributes(file, BasicFileAttributes.class); | ||
| } catch (FileNotFoundException e) { | ||
| continue; | ||
| } | ||
| if (attrs.isRegularFile()) { |
There was a problem hiding this comment.
This won't help us I think? We still want the file size two lines below where we'd have to make another call to the FS that we'd have to guard against "file not found".
| try { | ||
| attrs = Files.readAttributes(file, BasicFileAttributes.class); | ||
| } catch (FileNotFoundException e) { | ||
| continue; |
| final BasicFileAttributes attrs; | ||
| try { | ||
| attrs = Files.readAttributes(file, BasicFileAttributes.class); | ||
| } catch (FileNotFoundException e) { |
|
Thanks Yannick! |
original-brownbear
added a commit
to original-brownbear/elasticsearch
that referenced
this pull request
Nov 15, 2019
…lastic#49142) * Make FsBlobContainer Listing Resilient to Concurrent Modifications If we list out files in a folder via the lazily computed directory stream, we have to deal with concurrent deletes when reading the file attributes since we don't have a lock on the directory in any way. Closes elastic#37581
original-brownbear
added a commit
to original-brownbear/elasticsearch
that referenced
this pull request
Nov 15, 2019
…lastic#49142) * Make FsBlobContainer Listing Resilient to Concurrent Modifications If we list out files in a folder via the lazily computed directory stream, we have to deal with concurrent deletes when reading the file attributes since we don't have a lock on the directory in any way. Closes elastic#37581
original-brownbear
added a commit
that referenced
this pull request
Nov 15, 2019
…49142) (#49176) * Make FsBlobContainer Listing Resilient to Concurrent Modifications If we list out files in a folder via the lazily computed directory stream, we have to deal with concurrent deletes when reading the file attributes since we don't have a lock on the directory in any way. Closes #37581
original-brownbear
added a commit
that referenced
this pull request
Nov 15, 2019
…49142) (#49177) * Make FsBlobContainer Listing Resilient to Concurrent Modifications If we list out files in a folder via the lazily computed directory stream, we have to deal with concurrent deletes when reading the file attributes since we don't have a lock on the directory in any way. Closes #37581
This was referenced Feb 3, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
If we list out files in a folder via the lazily computed directory
stream, we have to deal with concurrent deletes when reading the file
attributes since we don't have a lock on the directory in any way.
Closes #37581