Add job table to state API #5076

pcmoritz · 2019-07-01T05:29:29Z

What do these changes do?

This exposes a list of jobs that are running or ran the Ray cluster to the global state API.

Related issue number

Linter

I've run scripts/format.sh to lint the changes in this PR.

AmplabJenkins · 2019-07-01T06:50:58Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/14991/
Test FAILed.

AmplabJenkins · 2019-07-01T08:31:04Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/14990/
Test PASSed.

AmplabJenkins · 2019-07-01T09:04:04Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1415/
Test FAILed.

edoakes · 2019-07-01T11:22:10Z

python/ray/state.py

+         Args:
+            job_id: A job ID to get information about.
+         Returns:
+            A dictionary with information about the job ID in question.


I think it would be better to define a JobInfo class instead of using a dictionary for this. It's only a few fields and no logic right now, but this will likely grow in scope over time. Would also improve documentation over just having the fields in a header comment for another function.

Yeah I agree! Let's do this change in a followup PR, since all the other tables (tasks, objects, clients) are dictionaries in the same format at the moment and we should be consistent.

Other possible choices we could consider are something json compatible with json-schema (would make it easy to put this API behind a REST endpoint in the future), protobuf, namedtuple (meh), dataclass (meh bc might require new python), any thoughts? cc @simon-mo

Few considerations:

there's a backport for dataclass pip install dataclasses so that won't be an issue.

in the api schema world, openapi schema is fairly big as well. (Kubernetes use it)

protobuf in python is hard to work with. Non-intuitive API, slow serialization

dataclass > namedtuple

edoakes · 2019-07-01T11:23:04Z

python/ray/state.py

+         Returns:
+            A dictionary with information about the job ID in question.
+        """
+        # Allow the argument to be either a DriverID or a hex string.


Did you mean JobID?

This isn't actually fixed in the comment above, right?

yeah, the fix got rolled back when I force pushed

edoakes · 2019-07-01T11:26:07Z

python/ray/state.py

+        """Fetch and parse the job table information for a single job ID.
+
+         Args:
+            job_id: A job ID to get information about.


Should note that it can be a JobID or hex.

not actually fixed, right?

yeah, it got rolled back

edoakes · 2019-07-01T11:29:15Z

python/ray/state.py

+    def job_table(self):
+        """Fetch and parse the Redis job table.
+         Returns:
+            Information about the Ray jobs in the cluster.


Information in what format?

AmplabJenkins · 2019-07-01T11:56:49Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1416/
Test FAILed.

jovany-wang · 2019-07-02T16:02:39Z

python/ray/state.py

+        """
+        # Allow the argument to be either a DriverID or a hex string.
+        if not isinstance(job_id, ray.JobID):
+            job_id = ray.JobID(hex_to_binary(job_id))


Also assert the type of job_id is str?

jovany-wang · 2019-07-02T16:08:03Z

src/ray/protobuf/gcs.proto

@@ -227,6 +227,12 @@ message JobTableData {
  bytes job_id = 1;
  // Whether it's dead.
  bool is_dead = 2;
+  // The UNIX timestamp corresponding to this event (job added or removed).
+  int64 timestamp = 3;


It's better to declare 2 fields start_timestamp and stop_timestamp explicitly instead of 1 timestamp.

Each entry in the JobTable is a log entry (either job addition, or job removal, if is_dead = true), so a single timestamp is indeed the more natural representation here. They get aggregated into start time and stop time in the client API.

jovany-wang · 2019-07-02T16:08:24Z

src/ray/protobuf/gcs.proto

+  // The UNIX timestamp corresponding to this event (job added or removed).
+  int64 timestamp = 3;
+  // IP of the node this job was started on.
+  string node_manager_address = 4;


just node address?

this is to be consistent with the ClientTable

AmplabJenkins · 2019-07-02T20:42:49Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15029/
Test PASSed.

AmplabJenkins · 2019-07-03T01:21:38Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15037/
Test FAILed.

AmplabJenkins · 2019-07-03T06:18:24Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15050/
Test PASSed.

AmplabJenkins · 2019-07-04T04:49:01Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1447/
Test FAILed.

AmplabJenkins · 2019-07-04T07:30:41Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1448/
Test FAILed.

AmplabJenkins · 2019-07-04T20:58:17Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1454/
Test FAILed.

pcmoritz · 2019-07-05T23:00:29Z

src/ray/raylet/node_manager.cc

    worker_pool_.RegisterDriver(std::move(worker));
    local_queues_.AddDriverTaskId(driver_task_id);
+    RAY_CHECK_OK(
+      gcs_client_->job_table().AppendJobData(JobID(driver_id),


At the moment, these definitions are pretty inconsistent.

This will get nicer once #5110 is merged.

AmplabJenkins · 2019-07-05T23:57:41Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15121/
Test FAILed.

python/ray/state.py

robertnishihara · 2019-07-06T00:16:52Z

python/ray/state.py

+         Returns:
+            A dictionary with information about the job ID in question.
+        """
+        # Allow the argument to be either a DriverID or a hex string.


This isn't actually fixed in the comment above, right?

robertnishihara · 2019-07-06T00:17:31Z

python/ray/state.py

+        """Fetch and parse the job table information for a single job ID.
+
+         Args:
+            job_id: A job ID to get information about.


not actually fixed, right?

robertnishihara · 2019-07-06T00:19:06Z

python/ray/state.py

+
+    def job_table(self):
+        """Fetch and parse the Redis job table.
+         Returns:


indentation is wrong and missing newline

robertnishihara · 2019-07-06T00:19:57Z

python/ray/state.py

+         Returns:
+            Information about the Ray jobs in the cluster,
+            namely a list of dicts with keys:
+            - "JobID" (sha1 identifier for the job),


Suggested change

- "JobID" (sha1 identifier for the job),

- "JobID" (identifier for the job),

It's just random bytes, and will probably be shortened soon.

robertnishihara · 2019-07-06T00:23:14Z

python/ray/tests/test_basic.py

@@ -2432,6 +2432,9 @@ def test_global_state_api(shutdown_only):
    with pytest.raises(Exception):
        ray.nodes()

+    with pytest.raises(Exception):
+        ray.jobs()


This should be changed to make sure it actually raises the error message we expect.

with pytest.raises(Exception, match="The ray global state API cannot be used before ray.init has been called."):

robertnishihara · 2019-07-06T00:25:20Z

src/ray/gcs/tables.cc

@@ -475,10 +475,14 @@ std::string ProfileTable::DebugString() const {
  return Log<UniqueID, ProfileTableData>::DebugString();
 }

-Status JobTable::AppendJobData(const JobID &job_id, bool is_dead) {
+Status JobTable::AppendJobData(const JobID &job_id, bool is_dead, int64_t timestamp,


Could just compute the timestamp inside this method instead of passing it in, but I don't really have a preference.

I prefer to have it outside, that makes it possible for somebody to use this function to use this function with a different timestamp than the current time.

Co-Authored-By: Robert Nishihara <[email protected]>

robertnishihara

Looks good to me assuming tests pass.

AmplabJenkins · 2019-07-06T01:59:52Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15125/
Test PASSed.

AmplabJenkins · 2019-07-06T03:54:32Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1468/
Test FAILed.

AmplabJenkins · 2019-07-06T04:16:03Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15130/
Test FAILed.

AmplabJenkins · 2019-07-06T05:00:09Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15132/
Test FAILed.

AmplabJenkins · 2019-07-06T05:10:20Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15134/
Test FAILed.

AmplabJenkins · 2019-07-06T06:19:21Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15137/
Test FAILed.

AmplabJenkins · 2019-07-06T06:28:15Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15138/
Test PASSed.

AmplabJenkins · 2019-07-06T07:12:26Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15139/
Test PASSed.

AmplabJenkins · 2019-07-06T12:11:20Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1472/
Test FAILed.

AmplabJenkins · 2019-07-06T20:30:35Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1477/
Test FAILed.

AmplabJenkins · 2019-07-06T23:20:39Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1479/
Test FAILed.

AmplabJenkins · 2019-07-07T02:04:29Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1481/
Test FAILed.

AmplabJenkins · 2019-07-07T07:22:51Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1484/
Test FAILed.

AmplabJenkins · 2019-07-07T07:23:08Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1485/
Test FAILed.

AmplabJenkins · 2019-07-07T07:23:26Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1486/
Test FAILed.

pcmoritz added 2 commits June 30, 2019 22:20

implement job table

746bf37

add docs

97525d5

pcmoritz mentioned this pull request Jul 1, 2019

Add driver table to state API #4940

Closed

1 task

update

00461d6

edoakes reviewed Jul 1, 2019

View reviewed changes

jovany-wang reviewed Jul 2, 2019

View reviewed changes

pcmoritz added 2 commits July 2, 2019 10:48

WTF

54efe09

Merge branch 'master' into job-table

523239c

pcmoritz mentioned this pull request Jul 3, 2019

Refine RegisterClientRequest message to make it clearer. #5057

Merged

update

854300b

pcmoritz force-pushed the job-table branch from 0366ede to 854300b Compare July 5, 2019 20:49

update

6c1749d

pcmoritz commented Jul 5, 2019

View reviewed changes

robertnishihara reviewed Jul 6, 2019

View reviewed changes

pcmoritz and others added 3 commits July 5, 2019 17:30

Update python/ray/state.py

2450585

Co-Authored-By: Robert Nishihara <[email protected]>

update

f460176

update

04c02ac

robertnishihara approved these changes Jul 6, 2019

View reviewed changes

update

8f3458d

linting

6a6f331

lint

67ca803

robertnishihara merged commit c5253cc into ray-project:master Jul 6, 2019

robertnishihara deleted the job-table branch July 6, 2019 07:05

	- "JobID" (sha1 identifier for the job),
	- "JobID" (identifier for the job),

Add job table to state API #5076

Add job table to state API #5076

Conversation

pcmoritz commented Jul 1, 2019

What do these changes do?

Related issue number

Linter

AmplabJenkins commented Jul 1, 2019

AmplabJenkins commented Jul 1, 2019

AmplabJenkins commented Jul 1, 2019

Choose a reason for hiding this comment

pcmoritz Jul 2, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AmplabJenkins commented Jul 1, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AmplabJenkins commented Jul 2, 2019

AmplabJenkins commented Jul 3, 2019

AmplabJenkins commented Jul 3, 2019

AmplabJenkins commented Jul 4, 2019

AmplabJenkins commented Jul 4, 2019

AmplabJenkins commented Jul 4, 2019

pcmoritz Jul 5, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AmplabJenkins commented Jul 5, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robertnishihara left a comment

Choose a reason for hiding this comment

AmplabJenkins commented Jul 6, 2019

AmplabJenkins commented Jul 6, 2019

AmplabJenkins commented Jul 6, 2019

AmplabJenkins commented Jul 6, 2019

AmplabJenkins commented Jul 6, 2019

AmplabJenkins commented Jul 6, 2019

AmplabJenkins commented Jul 6, 2019

AmplabJenkins commented Jul 6, 2019

AmplabJenkins commented Jul 6, 2019

AmplabJenkins commented Jul 6, 2019

AmplabJenkins commented Jul 6, 2019

AmplabJenkins commented Jul 7, 2019

AmplabJenkins commented Jul 7, 2019

AmplabJenkins commented Jul 7, 2019

AmplabJenkins commented Jul 7, 2019

pcmoritz Jul 2, 2019 •

edited

Loading

pcmoritz Jul 5, 2019 •

edited

Loading