[observability][export-api] Write actor events #47303

nikitavemuri · 2024-08-23T20:35:43Z

Why are these changes needed?

Write actor events to file as part of the export API. This logic is only run if RayConfig::instance().enable_export_api_write() is true. Default value is false.
Event write is called whenever a value in the actor event data schema is modified. Typically this occurs before writing ActorTableData to the GCS table or publishing the data for the dashboard

Related issue number

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Signed-off-by: Nikita Vemuri <[email protected]>

rkooo567

Code looks good!

can you run name:many_actors and many_nodes_actor_test_on_v2 from release tests with the flag on?

rkooo567 · 2024-08-26T17:49:59Z

src/ray/gcs/gcs_server/gcs_actor_manager.cc

@@ -224,6 +224,37 @@ const rpc::ActorTableData &GcsActor::GetActorTableData() const {

 rpc::ActorTableData *GcsActor::GetMutableActorTableData() { return &actor_table_data_; }

+void GcsActor::WriteActorExportEvent() const {
+  /// Write actor_table_data_ as a export actor event if


very nit; but I prefer this style to avoid nested code;

if (!RayConfig::instance().enable_export_api_write()) { return } ....

rkooo567 · 2024-08-26T17:55:26Z

src/ray/gcs/gcs_server/gcs_server_main.cc

@@ -89,7 +89,8 @@ int main(int argc, char *argv[]) {
  // Initialize event framework.
  if (RayConfig::instance().event_log_reporter_enabled() && !log_dir.empty()) {
    const std::vector<ray::SourceTypeVariant> source_types = {
-        ray::rpc::Event_SourceType::Event_SourceType_GCS};
+        ray::rpc::Event_SourceType::Event_SourceType_GCS,
+        ray::rpc::ExportEvent_SourceType_EXPORT_ACTOR};


can you add comment why this source type is set?

Signed-off-by: Nikita Vemuri <[email protected]>

nikitavemuri · 2024-08-27T15:46:37Z

can you run name:many_actors and many_nodes_actor_test_on_v2 from release tests with the flag on?

Yeah, here's the test run https://buildkite.com/ray-project/release/builds/21701

rkooo567 · 2024-08-28T20:27:12Z

@nikitavemuri did you run the test with the flag on?

Signed-off-by: Nikita Vemuri <[email protected]>

rkooo567

LGTM! Likely this can cause perf issue at large scale, and we will follow up perf after merging initial PRs

can-anyscale · 2024-09-06T00:44:15Z

Broke #47511 and is blocking release. I'm reverting to unblock.

This reverts commit 99af819.

Reverts #47303

Write actor events to file as part of the export API. This logic is only run if RayConfig::instance().enable_export_api_write() is true. Default value is false. Event write is called whenever a value in the actor event data schema is modified. Typically this occurs before writing ActorTableData to the GCS table or publishing the data for the dashboard Signed-off-by: ujjawal-khare <[email protected]>

…47516) Reverts ray-project#47303 Signed-off-by: ujjawal-khare <[email protected]>

- Add back code changes from [observability][export-api] Write actor events ray-project#47303 - Separate out actor manager export event test into a separate file so we can skip on windows. Update BUILD rule so all tests in src/ray/gcs/gcs_server/test/export_api are skipped on windows Signed-off-by: Nikita Vemuri <[email protected]> Co-authored-by: Nikita Vemuri <[email protected]> Signed-off-by: ujjawal-khare <[email protected]>

Write actor events to file as part of the export API. This logic is only run if RayConfig::instance().enable_export_api_write() is true. Default value is false. Event write is called whenever a value in the actor event data schema is modified. Typically this occurs before writing ActorTableData to the GCS table or publishing the data for the dashboard Signed-off-by: ujjawal-khare <[email protected]>

…47516) Reverts ray-project#47303 Signed-off-by: ujjawal-khare <[email protected]>

- Add back code changes from [observability][export-api] Write actor events ray-project#47303 - Separate out actor manager export event test into a separate file so we can skip on windows. Update BUILD rule so all tests in src/ray/gcs/gcs_server/test/export_api are skipped on windows Signed-off-by: Nikita Vemuri <[email protected]> Co-authored-by: Nikita Vemuri <[email protected]> Signed-off-by: ujjawal-khare <[email protected]>

Write actor events to file as part of the export API. This logic is only run if RayConfig::instance().enable_export_api_write() is true. Default value is false. Event write is called whenever a value in the actor event data schema is modified. Typically this occurs before writing ActorTableData to the GCS table or publishing the data for the dashboard Signed-off-by: ujjawal-khare <[email protected]>

…47516) Reverts ray-project#47303 Signed-off-by: ujjawal-khare <[email protected]>

- Add back code changes from [observability][export-api] Write actor events ray-project#47303 - Separate out actor manager export event test into a separate file so we can skip on windows. Update BUILD rule so all tests in src/ray/gcs/gcs_server/test/export_api are skipped on windows Signed-off-by: Nikita Vemuri <[email protected]> Co-authored-by: Nikita Vemuri <[email protected]> Signed-off-by: ujjawal-khare <[email protected]>

Write actor events to file as part of the export API. This logic is only run if RayConfig::instance().enable_export_api_write() is true. Default value is false. Event write is called whenever a value in the actor event data schema is modified. Typically this occurs before writing ActorTableData to the GCS table or publishing the data for the dashboard Signed-off-by: ujjawal-khare <[email protected]>

…47516) Reverts ray-project#47303 Signed-off-by: ujjawal-khare <[email protected]>

- Add back code changes from [observability][export-api] Write actor events ray-project#47303 - Separate out actor manager export event test into a separate file so we can skip on windows. Update BUILD rule so all tests in src/ray/gcs/gcs_server/test/export_api are skipped on windows Signed-off-by: Nikita Vemuri <[email protected]> Co-authored-by: Nikita Vemuri <[email protected]> Signed-off-by: ujjawal-khare <[email protected]>

Write actor events to file as part of the export API. This logic is only run if RayConfig::instance().enable_export_api_write() is true. Default value is false. Event write is called whenever a value in the actor event data schema is modified. Typically this occurs before writing ActorTableData to the GCS table or publishing the data for the dashboard Signed-off-by: ujjawal-khare <[email protected]>

…47516) Reverts ray-project#47303 Signed-off-by: ujjawal-khare <[email protected]>

- Add back code changes from [observability][export-api] Write actor events ray-project#47303 - Separate out actor manager export event test into a separate file so we can skip on windows. Update BUILD rule so all tests in src/ray/gcs/gcs_server/test/export_api are skipped on windows Signed-off-by: Nikita Vemuri <[email protected]> Co-authored-by: Nikita Vemuri <[email protected]> Signed-off-by: ujjawal-khare <[email protected]>

Write actor events to file as part of the export API. This logic is only run if RayConfig::instance().enable_export_api_write() is true. Default value is false. Event write is called whenever a value in the actor event data schema is modified. Typically this occurs before writing ActorTableData to the GCS table or publishing the data for the dashboard Signed-off-by: ujjawal-khare <[email protected]>

…47516) Reverts ray-project#47303 Signed-off-by: ujjawal-khare <[email protected]>

- Add back code changes from [observability][export-api] Write actor events ray-project#47303 - Separate out actor manager export event test into a separate file so we can skip on windows. Update BUILD rule so all tests in src/ray/gcs/gcs_server/test/export_api are skipped on windows Signed-off-by: Nikita Vemuri <[email protected]> Co-authored-by: Nikita Vemuri <[email protected]> Signed-off-by: ujjawal-khare <[email protected]>

Write actor events to file as part of the export API. This logic is only run if RayConfig::instance().enable_export_api_write() is true. Default value is false. Event write is called whenever a value in the actor event data schema is modified. Typically this occurs before writing ActorTableData to the GCS table or publishing the data for the dashboard Signed-off-by: ujjawal-khare <[email protected]>

…47516) Reverts ray-project#47303 Signed-off-by: ujjawal-khare <[email protected]>

- Add back code changes from [observability][export-api] Write actor events ray-project#47303 - Separate out actor manager export event test into a separate file so we can skip on windows. Update BUILD rule so all tests in src/ray/gcs/gcs_server/test/export_api are skipped on windows Signed-off-by: Nikita Vemuri <[email protected]> Co-authored-by: Nikita Vemuri <[email protected]> Signed-off-by: ujjawal-khare <[email protected]>

Write actor events to file as part of the export API. This logic is only run if RayConfig::instance().enable_export_api_write() is true. Default value is false. Event write is called whenever a value in the actor event data schema is modified. Typically this occurs before writing ActorTableData to the GCS table or publishing the data for the dashboard Signed-off-by: ujjawal-khare <[email protected]>

…47516) Reverts ray-project#47303 Signed-off-by: ujjawal-khare <[email protected]>

- Add back code changes from [observability][export-api] Write actor events ray-project#47303 - Separate out actor manager export event test into a separate file so we can skip on windows. Update BUILD rule so all tests in src/ray/gcs/gcs_server/test/export_api are skipped on windows Signed-off-by: Nikita Vemuri <[email protected]> Co-authored-by: Nikita Vemuri <[email protected]> Signed-off-by: ujjawal-khare <[email protected]>

Write actor events to file as part of the export API. This logic is only run if RayConfig::instance().enable_export_api_write() is true. Default value is false. Event write is called whenever a value in the actor event data schema is modified. Typically this occurs before writing ActorTableData to the GCS table or publishing the data for the dashboard Signed-off-by: ujjawal-khare <[email protected]>

…47516) Reverts ray-project#47303 Signed-off-by: ujjawal-khare <[email protected]>

- Add back code changes from [observability][export-api] Write actor events ray-project#47303 - Separate out actor manager export event test into a separate file so we can skip on windows. Update BUILD rule so all tests in src/ray/gcs/gcs_server/test/export_api are skipped on windows Signed-off-by: Nikita Vemuri <[email protected]> Co-authored-by: Nikita Vemuri <[email protected]> Signed-off-by: ujjawal-khare <[email protected]>

Write actor events to file as part of the export API. This logic is only run if RayConfig::instance().enable_export_api_write() is true. Default value is false. Event write is called whenever a value in the actor event data schema is modified. Typically this occurs before writing ActorTableData to the GCS table or publishing the data for the dashboard Signed-off-by: ujjawal-khare <[email protected]>

…47516) Reverts ray-project#47303 Signed-off-by: ujjawal-khare <[email protected]>

- Add back code changes from [observability][export-api] Write actor events ray-project#47303 - Separate out actor manager export event test into a separate file so we can skip on windows. Update BUILD rule so all tests in src/ray/gcs/gcs_server/test/export_api are skipped on windows Signed-off-by: Nikita Vemuri <[email protected]> Co-authored-by: Nikita Vemuri <[email protected]> Signed-off-by: ujjawal-khare <[email protected]>

Nikita Vemuri added 6 commits August 23, 2024 10:24

updates to event

8b1e93e

Signed-off-by: Nikita Vemuri <[email protected]>

write to file

d1d81c2

Signed-off-by: Nikita Vemuri <[email protected]>

add test

105a7f1

Signed-off-by: Nikita Vemuri <[email protected]>

add ff

659687a

Signed-off-by: Nikita Vemuri <[email protected]>

test util fn

35516d0

Signed-off-by: Nikita Vemuri <[email protected]>

formatting

13a73d1

Signed-off-by: Nikita Vemuri <[email protected]>

nikitavemuri added the go add ONLY when ready to merge, run all tests label Aug 23, 2024

move util fn

996b847

Signed-off-by: Nikita Vemuri <[email protected]>

nikitavemuri marked this pull request as ready for review August 26, 2024 17:31

nikitavemuri requested a review from a team as a code owner August 26, 2024 17:31

nikitavemuri requested a review from rkooo567 August 26, 2024 17:31

rkooo567 reviewed Aug 26, 2024

View reviewed changes

update

e859114

Signed-off-by: Nikita Vemuri <[email protected]>

rkooo567 added the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Aug 28, 2024

Nikita Vemuri added 6 commits August 28, 2024 14:04

merge

66f0cba

Signed-off-by: Nikita Vemuri <[email protected]>

merge

0c74f56

Signed-off-by: Nikita Vemuri <[email protected]>

fix merge

42357f5

Signed-off-by: Nikita Vemuri <[email protected]>

run tests

e45de2a

Signed-off-by: Nikita Vemuri <[email protected]>

merge

120f2c1

Signed-off-by: Nikita Vemuri <[email protected]>

fix merge

6ef355e

Signed-off-by: Nikita Vemuri <[email protected]>

nikitavemuri assigned rkooo567 Sep 5, 2024

rkooo567 approved these changes Sep 5, 2024

View reviewed changes

rkooo567 merged commit 99af819 into ray-project:master Sep 5, 2024
5 checks passed

can-anyscale added a commit that referenced this pull request Sep 6, 2024

Revert "[observability][export-api] Write actor events (#47303)"

1aad280

This reverts commit 99af819.

can-anyscale mentioned this pull request Sep 6, 2024

Revert "[observability][export-api] Write actor events" #47516

Merged

can-anyscale added a commit that referenced this pull request Sep 6, 2024

Revert "[observability][export-api] Write actor events" (#47516)

f336f51

Reverts #47303

nikitavemuri mentioned this pull request Sep 6, 2024

[observability][export-api] Write actor events #47529

Merged

8 tasks

ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024

Revert "[observability][export-api] Write actor events" (ray-project#…

20caf6f

…47516) Reverts ray-project#47303 Signed-off-by: ujjawal-khare <[email protected]>

ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024

Revert "[observability][export-api] Write actor events" (ray-project#…

c74a9b9

…47516) Reverts ray-project#47303 Signed-off-by: ujjawal-khare <[email protected]>

ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024

Revert "[observability][export-api] Write actor events" (ray-project#…

4ce9f12

…47516) Reverts ray-project#47303 Signed-off-by: ujjawal-khare <[email protected]>

ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024

Revert "[observability][export-api] Write actor events" (ray-project#…

d3b2744

…47516) Reverts ray-project#47303 Signed-off-by: ujjawal-khare <[email protected]>

ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024

Revert "[observability][export-api] Write actor events" (ray-project#…

eb79da0

…47516) Reverts ray-project#47303 Signed-off-by: ujjawal-khare <[email protected]>

ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024

Revert "[observability][export-api] Write actor events" (ray-project#…

43cd8f6

…47516) Reverts ray-project#47303 Signed-off-by: ujjawal-khare <[email protected]>

ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024

Revert "[observability][export-api] Write actor events" (ray-project#…

4d96599

…47516) Reverts ray-project#47303 Signed-off-by: ujjawal-khare <[email protected]>

ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024

Revert "[observability][export-api] Write actor events" (ray-project#…

ff4fa08

…47516) Reverts ray-project#47303 Signed-off-by: ujjawal-khare <[email protected]>

ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024

Revert "[observability][export-api] Write actor events" (ray-project#…

ad8b1b0

…47516) Reverts ray-project#47303 Signed-off-by: ujjawal-khare <[email protected]>

ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024

Revert "[observability][export-api] Write actor events" (ray-project#…

184e293

…47516) Reverts ray-project#47303 Signed-off-by: ujjawal-khare <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[observability][export-api] Write actor events #47303

[observability][export-api] Write actor events #47303

nikitavemuri commented Aug 23, 2024

rkooo567 left a comment

rkooo567 Aug 26, 2024

rkooo567 Aug 26, 2024

nikitavemuri commented Aug 27, 2024

rkooo567 commented Aug 28, 2024

rkooo567 left a comment

can-anyscale commented Sep 6, 2024

[observability][export-api] Write actor events #47303

[observability][export-api] Write actor events #47303

Conversation

nikitavemuri commented Aug 23, 2024

Why are these changes needed?

Related issue number

Checks

rkooo567 left a comment

Choose a reason for hiding this comment

rkooo567 Aug 26, 2024

Choose a reason for hiding this comment

rkooo567 Aug 26, 2024

Choose a reason for hiding this comment

nikitavemuri commented Aug 27, 2024

rkooo567 commented Aug 28, 2024

rkooo567 left a comment

Choose a reason for hiding this comment

can-anyscale commented Sep 6, 2024