[data] fix a race condition issue in OpBufferQueue #43015
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Hao Chen <[email protected]>
raulchen
requested review from
ericl,
scv119,
c21,
amogkam,
scottjlee,
bveeramani,
stephanie-wang and
Zandew
as code owners
Signed-off-by: Hao Chen <[email protected]>
Signed-off-by: Hao Chen <[email protected]>
raulchen
changed the title
c21
approved these changes
Feb 6, 2024
| ref = self._queue.popleft() | ||
| self._outputs_by_split[ref.output_split_idx].append(ref) | ||
| try: | ||
| ret = split_queue.popleft() |
There was a problem hiding this comment.
just one question - why we do not need lock here?
There was a problem hiding this comment.
dequeis thread-safe. 2) there will be at most one thread accessing this queue.
Previously the issue is because the length of self._queue may have changed when accessing self._queue[i]
There was a problem hiding this comment.
Got it. Can you add one comment for it?
lee1258561
pushed a commit
to pinterest/ray
that referenced
this pull request
Apr 4, 2024
…ject#43015) --------- Signed-off-by: Hao Chen <[email protected]>
8 tasks
lee1258561
added a commit
to pinterest/ray
that referenced
this pull request
Apr 10, 2024
…ject#43015) (#2) --------- Backporting ray-project#43015 to fix: IndexError: deque index out of range Build PR against OSS: ray-project#44469 <!-- Please add a reviewer to the assignee section when you create a PR. If you don't have the access to it, we will shortly find a reviewer and assign them to your PR. --> ## Why are these changes needed? <!-- Please give a short summary of the change and the problem this solves. --> ## Related issue number <!-- For example: "Closes ray-project#1234" --> ## Checks - [ ] I've signed off every commit(by using the -s flag, i.e., `git commit -s`) in this PR. - [ ] I've run `scripts/format.sh` to lint the changes in this PR. - [ ] I've included any doc changes needed for https://docs.ray.io/en/master/. - [ ] I've added any new APIs to the API Reference. For example, if I added a method in Tune, I've added it in `doc/source/tune/api/` under the corresponding `.rst` file. - [ ] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/ - Testing Strategy - [ ] Unit tests - [ ] Release tests - [ ] This PR is not tested :( Signed-off-by: Hao Chen <[email protected]> Co-authored-by: Hao Chen <[email protected]>
This was referenced Apr 18, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
Fix the following error.
This is a race condition bug introduced by #42601
This PR fixes the bug and also adds indexing to avoid inefficient linear scans.
Related issue number
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.