…tPipeline

…lineplan

Signed-off-by: Ubuntu <[email protected]>

Signed-off-by: Ubuntu <[email protected]>

My experiments (the script
) showed that dataset.split_at_indices() with SPREAD tasks have more predictable performance

Concretely: on 10 m5.4xlarge nodes with 5000 iops disk
calling ds.split_at_indices(81) on 200GB dataset with 400 blocks: the split_at_indices without this PR takes 7-19 seconds, split_at_indices with SPREAD takes 7-12 seconds.

Signed-off-by: Ubuntu <[email protected]>

…on (ray-project#26634)

Signed-off-by: Ubuntu <[email protected]>

…ject#26458)

Signed-off-by: Ubuntu <[email protected]>

Co-authored-by: Kourosh Hakhamaneshi <[email protected]>
Signed-off-by: Ubuntu <[email protected]>

…ject#26094)

Co-authored-by: Eric Liang <[email protected]>
Co-authored-by: matthewdeng <[email protected]>
Co-authored-by: Matthew Deng <[email protected]>
Co-authored-by: Richard Liaw <[email protected]>
Signed-off-by: Ubuntu <[email protected]>

…y-project#26643)

Signed-off-by: Ubuntu <[email protected]>

…project#26637)

Signed-off-by: Ubuntu <[email protected]>

…roject#26640)

Signed-off-by: Ubuntu <[email protected]>

…ints (ray-project#26641)

This PR replaces dataset.split(.., equal=True) implementation by dataset.split_at_indices() . My experiments (the script
) showed that dataset.split_at_indices() have more predictable performance than the dataset.split(…)

Concretely: on 10 m5.4xlarge nodes with 5000 iops disk

calling ds.split(81) on 200GB dataset with 400 blocks: the split takes 20-40 seconds, split_at_indices takes ~12 seconds.

calling ds.split(163) on 200GB dataset with 400 blocks, the split takes 40-100 seconds, split_at_indices takes ~24 seconds.

I don’t have much insight of dataset.split implementation, but with dataset.split_at_indices() we are just doing SPREAD to num_split_at_indices tasks, which yield much stable performance.

Note: clean up the usage of experimental locality_hints in ray-project#26647
Signed-off-by: Ubuntu <[email protected]>

Why are these changes needed?
Since locality_hints is an experimental feature, we stop promoting it in doc and don't enable it in AIR. See ray-project#26641 for more context

Signed-off-by: Ubuntu <[email protected]>

Signed-off-by: Ubuntu <[email protected]>

…ray-project#26428)

Signed-off-by: Ubuntu <[email protected]>

…project#26521)

Signed-off-by: Ubuntu <[email protected]>

- Stop using dot command to run ci.sh script: it doesn't fail the build if the command fails for windows and is generally dangerous since it will make unexpected changes to the current shell.
- Fix uncovered windows build issues.

Signed-off-by: Ubuntu <[email protected]>

Signed-off-by: Ubuntu <[email protected]>

Signed-off-by: Ubuntu <[email protected]>

…lineplan

…ipelineplan

Signed-off-by: Ubuntu <[email protected]>

Signed-off-by: jianoaix <[email protected]>

…lineplan

…is arg in ExecutionPlan instead of Stage