GetAllJobInfo is_running_tasks is not returning the correct value when driver starts ray

[sofianhnaide]

What happened + What you expected to happen

The is_running_tasks attribute on GetAllJobInfo is not returning the expected value (True) when no ray cluster is started, but it returns the correct value when a cluster is already started.

e.g.

import ray

@ray.remote
def f(x):
    import time
    time.sleep(5)
    return 0

ray.get([f.remote(i) for i in range(4)])

to reproduce:

1. Run ray stop -f
2. Run the script above and query the job API

Actual: while the task are in progress, the is_running_tasks flag should be True but it is False.
Expected: It should be True. If you repeat the steps above after pre-starting ray with ray start --head. The expected value is returned.

Versions / Dependencies

2.10

Reproduction script

import ray

@ray.remote
def f(x):
import time
time.sleep(5)
return 0

ray.get([f.remote(i) for i in range(4)])

Issue Severity

Medium: It is a significant difficulty but I can work around it.

[rynewang]

Can repro. The probe code:

import ray

ray.init()

def f():
    client = ray.worker.global_worker.gcs_client
    job_id = ray.worker.global_worker.current_job_id.binary()
    all_job_info = client.get_all_job_info()
    for job_id in all_job_info:
        print(job_id, all_job_info[job_id].is_running_tasks)
        
f()

It prints the task running code's status = False, if the cluster is started from within the driver. Should be True.

[rynewang]

So when there is a GetAllJobInfo call, the gcs job_manager calls the core worker client NumPendingTasks at gcs_job_manager.cc:220. When the job is started with ray start --head all is good; when the job starts from the job, we get this:

[2024-04-10 20:50:27,362 D 81581 11854183] (gcs_server) core_worker_client_pool.cc:39: Connected to worker 01000000ffffffffffffffffffffffffffffffffffffffffffffffff with address 127.0.0.1:0
[2024-04-10 20:50:27,362 D 81581 11854183] (gcs_server) gcs_job_manager.cc:220: Send NumPendingTasksRequest to worker 01000000ffffffffffffffffffffffffffffffffffffffffffffffff
[2024-04-10 20:50:27,363 D 81581 11854183] (gcs_server) gcs_job_manager.cc:226: Received NumPendingTasksReply from worker 01000000ffffffffffffffffffffffffffffffffffffffffffffffff
[2024-04-10 20:50:27,363 W 81581 11854183] (gcs_server) gcs_job_manager.cc:228: Failed to get is_running_tasks from core worker: GrpcUnavailable: RPC Error message: failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:0: Can't assign requested address; RPC Error details:
[2024-04-10 20:50:27,363 D 81581 11854183] (gcs_server) gcs_job_manager.cc:185: Finished getting all job info.

See the core worker address being 127.0.0.1:0. Will look more at where this is populated.