[Ray Client] Transfer dashboard_url over gRPC instead of ray.remote

[ckw017]

Why are these changes needed?

The ray.remote call is spawning worker tasks on the head node even if their client doesn't do anything, spawning unexpected workers.

Note: dashboard_url behavior is already tested by test_client_builder

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[ijrsvt]

LGTM!

[rkooo567]

can you add an unit test to verify connecting a new job won't create IDLE workers until it submits a first task?

[ckw017]

can you add an unit test to verify connecting a new job won't create IDLE workers until it submits a first task?

Is there an internal API I can use to check this easily? Even with the current behavior the IDLE worker gets cleaned up almost immediately, so its hard to tell when it happens without spam listing processes frequently

[rkooo567]

IDLE workers are not killed if the # of them < num_cpus

You can probably use this API ray.experimental.state.api.list_workers. Basically we can

1. connect
2. wait for some time
3. make sure no workers are started
4. submit a task
5. make sure # of idle workers are same as num_cpus

[ckw017]

@rkooo567 added a test, and confirmed that it fails without these changes (without changes, two worker processes spawn before we schedule any tasks)

[rkooo567]

This is awesome! Thanks for a quick fix @ckw017 !

[rkooo567]

Lots of dataset tests failed. Is it related?

[rkooo567]

seems like dataset failure is unrelated

[DmitriGekhtman]

How severe is the issue resolved here for users of Ray Client in Ray 2.2.0?

[ckw017]

How severe is the issue resolved here for users of Ray Client in Ray 2.2.0?

@rkooo567 is this worth picking into 2.2.0? IMO its low risk to add on if we're already planning other cherry picks anyway, but also think it can wait until the next release

[rkooo567]

I think it is not allowed to cherry pick non critical fixes. So let's just do it in the next release

[will-hang]

This is actually really critical for us and is causing some of our core services to fail due to OOMs in ray head.

When memory leaks accumulate in ray head that seem related to these issues:

• [Core] Memory leak (?) with no activity  #28047
• [Dashboard] The agent.py process leaks memory #29199

another IDLE worker spun up in ray head upon a Client connection will cause the pod to topple over, killing our service. We're looking forward to upgrading to 2.2 to hopefully resolve this.

Given that the change is so important and appears small, is there a way we could get it in 2.2? Or will we have to wait until 2.3?

[DmitriGekhtman]

spawning unexpected workers

How many unexpected workers? What is the impact of the bug resolved here?