Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Core] Expose NodeDeathInfo in ActorDiedError #45497

Merged
merged 19 commits into from
Jun 1, 2024
Merged

[Core] Expose NodeDeathInfo in ActorDiedError #45497

merged 19 commits into from
Jun 1, 2024

Conversation

ruisearch42
Copy link
Contributor

@ruisearch42 ruisearch42 commented May 22, 2024 •
edited
Loading

Why are these changes needed?

This is the 4th PR of a series to better propagate and expose node death information.

This PR does the following:

  • Introduce NodeDeathInfo in ActorDeathCause so that node death information is exposed to the user
  • When the actor is dead, populate NodeDeathInfo from that obtained from GCS
  • To keep backwards compatibility, (1) keep the preempted flag in ActorDiedError, but derive this value through NodeDeathInfo; (2) keep the broadcast of ActorTableData when actor's node receives drain request.

Related issue number

[1/5] #45128
[2/5] #45357
[3/5] #45320

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@ruisearch42 ruisearch42 self-assigned this May 28, 2024
ruisearch42 and others added 4 commits May 29, 2024 18:34
@ruisearch42
update
6debb1d 
Signed-off-by: Rui Qiao <[email protected]>
@ruisearch42
update
5e07261 
Signed-off-by: Rui Qiao <[email protected]>
@ruisearch42
Merge branch 'master' into actor_died
61ad6f1
@ruisearch42
package name
b09fcad 
Signed-off-by: Rui Qiao <[email protected]>
@ruisearch42 ruisearch42 removed their assignment May 30, 2024
@ruisearch42 ruisearch42 added the go add ONLY when ready to merge, run all tests label May 30, 2024
@ruisearch42 ruisearch42 self-assigned this May 30, 2024
@ruisearch42
update
674c67c 
Signed-off-by: Rui Qiao <[email protected]>
@ruisearch42
minor
41042ef 
Signed-off-by: Rui Qiao <[email protected]>
@ruisearch42 ruisearch42 marked this pull request as ready for review May 30, 2024 22:43
@ruisearch42 ruisearch42 requested review from a team, ericl, pcmoritz and raulchen as code owners May 30, 2024 22:43
@ruisearch42 ruisearch42 assigned jjyao and unassigned ruisearch42 May 30, 2024
jjyao
jjyao reviewed May 31, 2024
View reviewed changes
python/ray/exceptions.py Outdated Show resolved Hide resolved
src/ray/gcs/gcs_server/gcs_actor_manager.cc Outdated Show resolved Hide resolved
python/ray/exceptions.py Show resolved Hide resolved
@ruisearch42
comments
a4b040c 
Signed-off-by: Rui Qiao <[email protected]>
@ruisearch42
lint
893843a 
Signed-off-by: Rui Qiao <[email protected]>
@ruisearch42
add tests
fae2d54 
Signed-off-by: Rui Qiao <[email protected]>
@ruisearch42
update
86b6d56 
Signed-off-by: Rui Qiao <[email protected]>
@ruisearch42
update
925e9ae 
Signed-off-by: Rui Qiao <[email protected]>
jjyao
jjyao reviewed May 31, 2024
View reviewed changes
python/ray/exceptions.py Outdated Show resolved Hide resolved
src/ray/gcs/gcs_server/gcs_actor_manager.cc Outdated Show resolved Hide resolved
@ruisearch42 ruisearch42 mentioned this pull request May 31, 2024
Merged
8 tasks
@ruisearch42
update
9e8cf8f 
Signed-off-by: Rui Qiao <[email protected]>
ruisearch42 and others added 3 commits May 31, 2024 11:35
jjyao
jjyao reviewed May 31, 2024
View reviewed changes
Copy link
Collaborator

@jjyao jjyao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

python/ray/tests/test_node_death.py Outdated Show resolved Hide resolved
src/ray/core_worker/transport/direct_actor_task_submitter.cc Outdated Show resolved Hide resolved
src/ray/gcs/gcs_server/gcs_actor_manager.cc Outdated Show resolved Hide resolved
python/ray/tests/test_draining.py Outdated Show resolved Hide resolved
python/ray/tests/test_node_death.py Outdated Show resolved Hide resolved
@ruisearch42
comm
60dda09 
Signed-off-by: Rui Qiao <[email protected]>
@ruisearch42
minor
51c194a 
Signed-off-by: Rui Qiao <[email protected]>
@jjyao jjyao merged commit 7021b10 into ray-project:master Jun 1, 2024
6 checks passed
richardsliu pushed a commit to richardsliu/ray that referenced this pull request Jun 12, 2024
@ruisearch42 @richardsliu
[Core] Expose NodeDeathInfo in ActorDiedError (ray-project#45497)
135766a 
Signed-off-by: Rui Qiao <[email protected]>
Signed-off-by: Richard Liu <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jjyao jjyao jjyao approved these changes

@ericl ericl Awaiting requested review from ericl

@pcmoritz pcmoritz Awaiting requested review from pcmoritz pcmoritz is a code owner

@raulchen raulchen Awaiting requested review from raulchen raulchen is a code owner

Assignees

@jjyao jjyao

Labels
go add ONLY when ready to merge, run all tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@ruisearch42 @jjyao