[data] Enable per-op resource reservation #43171
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Hao Chen <[email protected]>
Signed-off-by: Hao Chen <[email protected]>
raulchen
requested review from
ericl,
scv119,
c21,
amogkam,
scottjlee,
bveeramani,
stephanie-wang and
omatthew98
as code owners
Signed-off-by: Hao Chen <[email protected]>
Signed-off-by: Hao Chen <[email protected]>
Signed-off-by: Hao Chen <[email protected]>
Signed-off-by: Hao Chen <[email protected]>
Signed-off-by: Hao Chen <[email protected]>
Signed-off-by: Hao Chen <[email protected]>
Signed-off-by: Hao Chen <[email protected]>
Signed-off-by: Hao Chen <[email protected]>
Signed-off-by: Hao Chen <[email protected]>
Signed-off-by: Hao Chen <[email protected]>
Signed-off-by: Hao Chen <[email protected]> fix usage_str Signed-off-by: Hao Chen <[email protected]>
Signed-off-by: Hao Chen <[email protected]>
Signed-off-by: Hao Chen <[email protected]>
raulchen
force-pushed
the
enable-memory-reservation
branch
from
3c3fd30
to
df2e4be
Compare
Signed-off-by: Hao Chen <[email protected]>
Signed-off-by: Hao Chen <[email protected]>
Signed-off-by: Hao Chen <[email protected]>
Signed-off-by: Hao Chen <[email protected]>
Signed-off-by: Hao Chen <[email protected]>
Signed-off-by: Hao Chen <[email protected]>
Signed-off-by: Hao Chen <[email protected]>
bveeramani
reviewed
Feb 26, 2024
Comment on lines
395
to
396
| ExecutionResources(cpu=1) as its incremental usage. | ||
| Args: |
There was a problem hiding this comment.
Suggested change
| ExecutionResources(cpu=1) as its incremental usage. | |
| Args: | |
| ExecutionResources(cpu=1) as its incremental usage. | |
| Args: |
Signed-off-by: Hao Chen <[email protected]>
Signed-off-by: Hao Chen <[email protected]>
Signed-off-by: Hao Chen <[email protected]>
Signed-off-by: Hao Chen <[email protected]>
Signed-off-by: Hao Chen <[email protected]>
Signed-off-by: Hao Chen <[email protected]>
Signed-off-by: Hao Chen <[email protected]>
Signed-off-by: Hao Chen <[email protected]>
bveeramani
approved these changes
Feb 27, 2024
can-anyscale
added a commit
that referenced
this pull request
Feb 28, 2024
This reverts commit d6380d4.
|
as well as this test 
|
8 tasks
can-anyscale
pushed a commit
that referenced
this pull request
Feb 29, 2024
Fix bugs in ReservationOpResourceAllocator (introduced by #43171) * We treat a map op and its following non-map ops as the same group. `update_resources` already handles this properly . But `_should_unblock_streaming_output_backpressure` and ` _op_outputs_reserved_remaining` didn't consider this. * Since we don't reserve any resources for `limit` and `streaming_split`, should set `num_cpus=0` for their tasks. * `_reserved_for_op_outputs` currently also includes op's internal output buffers. This is incorrect, because when `preserve_order=True`, task outputs will accumulate in op's internal output buffer, and use all the memory budget from `_reserved_for_op_outputs`. Then we still don't have memory budget to pull the blocks from the internal output buffer. Excluding the internal output buffer from `_reserved_for_op_outputs` fixes this issue. Also deflake `test_backpressure_from_output` and `test_e2e_autoscaling_up`, as they depend on physical memory size of the node. Signed-off-by: Hao Chen <[email protected]>
8 tasks
raulchen
added a commit
that referenced
this pull request
Feb 29, 2024
there is a perf regression for those 2 small-sized test cases in ray-data-resnet50-ingest-file-size-benchmark, due to the new backpressure change (#43171). Update some configs to fix the perf issue. Signed-off-by: Hao Chen <[email protected]>
8 tasks
raulchen
added a commit
that referenced
this pull request
Mar 4, 2024
…on not enabled (#43686) #43171 increased the default memory limit fraction to 50%. because with memory reservation, we have more precise control over the memory. This PR set the default back to 25%, when memory reservation is not enabled, to prevent regression. Signed-off-by: Hao Chen <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
This PR enables per-op resource reservation by default. See docstring of
ReservationOpResourceAllocatorfor the concrete protocol.Besides, this PR also:
OpResourceLimitertoOpResourceAllocator, and adds 2 new APIs.ReservationOpResourceAllocator.StreamingOutputBackpressurePolicy. as this feature is now implemented asReservationOpResourceAllocator.max_task_output_bytes_to_read.use_runtime_metrics_scheduling.ExecutionResources.Related issue number
Closes #42217
#40754
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.