Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[core] Fix the placement group stress test regression. #34192

Merged
merged 1 commit into from
Apr 10, 2023
Merged

[core] Fix the placement group stress test regression. #34192

merged 1 commit into from
Apr 10, 2023

Conversation

fishbone
Copy link
Contributor

@fishbone fishbone commented Apr 8, 2023 •
edited
Loading

Signed-off-by: Yi Cheng [email protected]

Why are these changes needed?

The regression is because of enabling ray syncer. In the code, whenever the pg is created and deleted, raylet will actively send a message to GCS and this introduced a lot of workload to the GCS and thus make the code run slow.
If disable ray syncer, raylet won't create this message and not sending it to GCS.

There is no need doing this since when new resource is added to local node, ray syncer will be able to notice this and the resource will be pushed to GCS after 100ms.

This PR deleted this logic and thus fix the regression.

before: placement group create/removal per second 1271.32 +- 8.27
after: placement group create/removal per second 1282.83 +- 3.99

For release test:

perf_metrics = [{'perf_metric_name': 'pgs_per_second', 'perf_metric_value': 17.061243668170643, 'perf_metric_type': 'THROUGHPUT'}, {'perf_metric_name': 'dashboard_p50_latency_ms', 'perf_metric_value': 3.261, 'perf_metric_type': 'LATENCY'}, {'perf_metric_name': 'dashboard_p95_latency_ms', 'perf_metric_value': 129.682, 'perf_metric_type': 'LATENCY'}, {'perf_metric_name': 'dashboard_p99_latency_ms', 'perf_metric_value': 141.648, 'perf_metric_type': 'LATENCY'}]

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@fishbone
fix
e0a2b1f 
Signed-off-by: Yi Cheng <[email protected]>
@fishbone fishbone added release-blocker P0 Issue that blocks the release Ray 2.4 labels Apr 8, 2023
@fishbone
Copy link
Contributor Author

fishbone commented Apr 8, 2023

Needs to run some pg test to ensure no regression in other cases.

This was linked to issues Apr 8, 2023
[Ray component: Core] Respect the placement group when num_gpus=0, num_cpus=0 #27931
Open
[core] perf regression : placement group create / removal #33948
Closed
@fishbone fishbone removed a link to an issue Apr 8, 2023
[Ray component: Core] Respect the placement group when num_gpus=0, num_cpus=0 #27931
Open
@fishbone fishbone linked an issue Apr 8, 2023 that may be closed by this pull request
[core] pgs_per_second regression #33925
Closed
scv119
scv119 approved these changes Apr 9, 2023
View reviewed changes
Copy link
Contributor

@scv119 scv119 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does the benchmark show this PR fix the regression?

@fishbone
Copy link
Contributor Author

@scv119 added the data into the description

@fishbone fishbone merged commit 4ad2cd1 into ray-project:master Apr 10, 2023
fishbone added a commit to fishbone/ray that referenced this pull request Apr 10, 2023
@fishbone
[core] Fix the placement group stress test regression. (ray-project#3…
7d749ac 
…4192)

Signed-off-by: Yi Cheng <[email protected]>


The regression is because of enabling ray syncer. In the code, whenever the pg is created and deleted, raylet will actively send a message to GCS and this introduced a lot of workload to the GCS and thus make the code run slow.
If disable ray syncer, raylet won't create this message and not sending it to GCS.

There is no need doing this since when new resource is added to local node, ray syncer will be able to notice this and the resource will be pushed to GCS after 100ms.

This PR deleted this logic and thus fix the regression.

```
before: placement group create/removal per second 1271.32 +- 8.27
after: placement group create/removal per second 1282.83 +- 3.99
```

For release test:
```
perf_metrics = [{'perf_metric_name': 'pgs_per_second', 'perf_metric_value': 17.061243668170643, 'perf_metric_type': 'THROUGHPUT'}, {'perf_metric_name': 'dashboard_p50_latency_ms', 'perf_metric_value': 3.261, 'perf_metric_type': 'LATENCY'}, {'perf_metric_name': 'dashboard_p95_latency_ms', 'perf_metric_value': 129.682, 'perf_metric_type': 'LATENCY'}, {'perf_metric_name': 'dashboard_p99_latency_ms', 'perf_metric_value': 141.648, 'perf_metric_type': 'LATENCY'}]
```
clarng pushed a commit that referenced this pull request Apr 11, 2023
@fishbone
[core] Fix the placement group stress test regression. (#34192) (#34249)
cb6f1c2 
Signed-off-by: Yi Cheng <[email protected]>


The regression is because of enabling ray syncer. In the code, whenever the pg is created and deleted, raylet will actively send a message to GCS and this introduced a lot of workload to the GCS and thus make the code run slow.
If disable ray syncer, raylet won't create this message and not sending it to GCS.

There is no need doing this since when new resource is added to local node, ray syncer will be able to notice this and the resource will be pushed to GCS after 100ms.

This PR deleted this logic and thus fix the regression.

```
before: placement group create/removal per second 1271.32 +- 8.27
after: placement group create/removal per second 1282.83 +- 3.99
```

For release test:
```
perf_metrics = [{'perf_metric_name': 'pgs_per_second', 'perf_metric_value': 17.061243668170643, 'perf_metric_type': 'THROUGHPUT'}, {'perf_metric_name': 'dashboard_p50_latency_ms', 'perf_metric_value': 3.261, 'perf_metric_type': 'LATENCY'}, {'perf_metric_name': 'dashboard_p95_latency_ms', 'perf_metric_value': 129.682, 'perf_metric_type': 'LATENCY'}, {'perf_metric_name': 'dashboard_p99_latency_ms', 'perf_metric_value': 141.648, 'perf_metric_type': 'LATENCY'}]
```
elliottower pushed a commit to elliottower/ray that referenced this pull request Apr 22, 2023
@fishbone @elliottower
[core] Fix the placement group stress test regression. (ray-project#3…
7ce3b0b 
…4192)

Signed-off-by: Yi Cheng <[email protected]>

The regression is because of enabling ray syncer. In the code, whenever the pg is created and deleted, raylet will actively send a message to GCS and this introduced a lot of workload to the GCS and thus make the code run slow.
If disable ray syncer, raylet won't create this message and not sending it to GCS.

There is no need doing this since when new resource is added to local node, ray syncer will be able to notice this and the resource will be pushed to GCS after 100ms.

This PR deleted this logic and thus fix the regression.

```
before: placement group create/removal per second 1271.32 +- 8.27
after: placement group create/removal per second 1282.83 +- 3.99
```

For release test:
```
perf_metrics = [{'perf_metric_name': 'pgs_per_second', 'perf_metric_value': 17.061243668170643, 'perf_metric_type': 'THROUGHPUT'}, {'perf_metric_name': 'dashboard_p50_latency_ms', 'perf_metric_value': 3.261, 'perf_metric_type': 'LATENCY'}, {'perf_metric_name': 'dashboard_p95_latency_ms', 'perf_metric_value': 129.682, 'perf_metric_type': 'LATENCY'}, {'perf_metric_name': 'dashboard_p99_latency_ms', 'perf_metric_value': 141.648, 'perf_metric_type': 'LATENCY'}]
```

Signed-off-by: elliottower <[email protected]>
ProjectsByJackHe pushed a commit to ProjectsByJackHe/ray that referenced this pull request May 4, 2023
@fishbone @ProjectsByJackHe
[core] Fix the placement group stress test regression. (ray-project#3…
a5a29c5 
…4192)

Signed-off-by: Yi Cheng <[email protected]>

The regression is because of enabling ray syncer. In the code, whenever the pg is created and deleted, raylet will actively send a message to GCS and this introduced a lot of workload to the GCS and thus make the code run slow.
If disable ray syncer, raylet won't create this message and not sending it to GCS.

There is no need doing this since when new resource is added to local node, ray syncer will be able to notice this and the resource will be pushed to GCS after 100ms.

This PR deleted this logic and thus fix the regression.

```
before: placement group create/removal per second 1271.32 +- 8.27
after: placement group create/removal per second 1282.83 +- 3.99
```

For release test:
```
perf_metrics = [{'perf_metric_name': 'pgs_per_second', 'perf_metric_value': 17.061243668170643, 'perf_metric_type': 'THROUGHPUT'}, {'perf_metric_name': 'dashboard_p50_latency_ms', 'perf_metric_value': 3.261, 'perf_metric_type': 'LATENCY'}, {'perf_metric_name': 'dashboard_p95_latency_ms', 'perf_metric_value': 129.682, 'perf_metric_type': 'LATENCY'}, {'perf_metric_name': 'dashboard_p99_latency_ms', 'perf_metric_value': 141.648, 'perf_metric_type': 'LATENCY'}]
```

Signed-off-by: Jack He <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@scv119 scv119 scv119 approved these changes

Assignees

@scv119 scv119

@rkooo567 rkooo567

Labels
Ray 2.4 release-blocker P0 Issue that blocks the release
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[core] perf regression : placement group create / removal [core] pgs_per_second regression
3 participants
@fishbone @scv119 @rkooo567