[direct call] changes raylet to push tasks to worker #5140

zhijunfu · 2019-07-08T05:00:24Z

What do these changes do?

The previous change adds a grpc server to worker, this pr changes raylet to assign tasks instead of waiting for workers to get tasks.

Related issue number

#5029

Linter

[Y] I've run scripts/format.sh to lint the changes in this PR.

zhijunfu · 2019-07-08T05:00:50Z

@raulchen @jiangzihao2009

AmplabJenkins · 2019-07-08T07:39:24Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15181/
Test PASSed.

src/ray/core_worker/task_execution.cc

src/ray/rpc/server_call.h

src/ray/raylet/node_manager.cc

src/ray/raylet/node_manager.h

src/ray/raylet/node_manager.cc

zhijunfu

Thanks. Updated accordingly.

src/ray/core_worker/task_execution.cc

src/ray/raylet/node_manager.cc

src/ray/raylet/node_manager.h

src/ray/rpc/server_call.h

src/ray/raylet/node_manager.cc

src/ray/core_worker/task_execution.cc

src/ray/raylet/node_manager.cc

src/ray/raylet/node_manager.h

AmplabJenkins · 2019-07-08T14:26:16Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15199/
Test PASSed.

AmplabJenkins · 2019-07-08T16:12:28Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15202/
Test PASSed.

src/ray/core_worker/core_worker.cc

src/ray/core_worker/mock_worker.cc

src/ray/core_worker/core_worker.h

src/ray/core_worker/task_execution.cc

src/ray/raylet/node_manager.cc

src/ray/raylet/node_manager.h

Co-Authored-By: Stephanie Wang <[email protected]>

AmplabJenkins · 2019-07-09T05:44:36Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15222/
Test FAILed.

raulchen · 2019-07-09T09:10:29Z

I just merged a PR. Can you resolve the conflicts?

AmplabJenkins · 2019-07-09T17:16:42Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1545/
Test FAILed.

AmplabJenkins · 2019-07-09T17:44:13Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1546/
Test FAILed.

stephanie-wang

Thanks, this is looking great! I left a couple comments that I would like addressed before we merge, but otherwise looks good.

src/ray/core_worker/transport/raylet_transport.cc

src/ray/raylet/node_manager.cc

zhijunfu · 2019-07-10T01:45:37Z

Thanks for reviewing! I think all comments are addressed.

AmplabJenkins · 2019-07-10T02:12:52Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1562/
Test FAILed.

AmplabJenkins · 2019-07-10T04:21:41Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15253/
Test PASSed.

zhijunfu · 2019-07-10T05:43:18Z

travis, retest this please

AmplabJenkins · 2019-07-10T06:40:25Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1570/
Test FAILed.

AmplabJenkins · 2019-07-10T07:20:57Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15260/
Test PASSed.

…er_grpc

AmplabJenkins · 2019-07-10T08:50:50Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1575/
Test FAILed.

AmplabJenkins · 2019-07-10T09:33:50Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15265/
Test PASSed.

AmplabJenkins · 2019-07-10T11:51:43Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1582/
Test FAILed.

AmplabJenkins · 2019-07-10T11:53:40Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15272/
Test PASSed.

AmplabJenkins · 2019-07-11T11:12:51Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15307/
Test FAILed.

AmplabJenkins · 2019-07-11T15:45:48Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1616/
Test FAILed.

AmplabJenkins · 2019-07-11T17:36:49Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15312/
Test PASSed.

AmplabJenkins · 2019-07-11T17:50:04Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1621/
Test FAILed.

stephanie-wang · 2019-07-11T18:01:47Z

I couldn't trigger a travis build for some reason...

AmplabJenkins · 2019-07-11T18:44:45Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1624/
Test FAILed.

AmplabJenkins · 2019-07-11T20:31:43Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15315/
Test PASSed.

zhijunfu added 12 commits June 24, 2019 16:02

refactor grpc server

7f635b4

format

2205c87

change GetTask() to PushTask()

0703b75

merge code, and fix test

6d5228a

merge

4c3eb6e

change PushTask to AssignTask

d1b20e5

format

1a663bc

merge

4ef00b2

add resource_ids

6caa126

move done_callback to server call

302e79f

remove SetTaskHandler and initialize it in task receiver's constructor

70d6e26

format

17b22e4

raulchen reviewed Jul 8, 2019

View reviewed changes

zhijunfu added 2 commits July 8, 2019 19:16

resolve comments

dbf666c

update

a5f301d

zhijunfu commented Jul 8, 2019

View reviewed changes

update

0c762f1

raulchen reviewed Jul 8, 2019

View reviewed changes

stephanie-wang self-assigned this Jul 8, 2019

stephanie-wang requested changes Jul 8, 2019

View reviewed changes

Update src/ray/core_worker/core_worker.cc

819614f

Co-Authored-By: Stephanie Wang <[email protected]>

resolve comments

beb01be

zhijunfu added 2 commits July 9, 2019 17:15

merge

be05ed6

format

878cae4

stephanie-wang approved these changes Jul 9, 2019

View reviewed changes

src/ray/core_worker/transport/raylet_transport.cc Outdated Show resolved Hide resolved

src/ray/raylet/node_manager.cc Show resolved Hide resolved

src/ray/raylet/node_manager.cc Outdated Show resolved Hide resolved

src/ray/raylet/node_manager.cc Show resolved Hide resolved

resolve comments

32b3d3e

zhijunfu added 2 commits July 10, 2019 12:20

fix build

d157b84

format

42179ea

Merge branch 'master' of https://github.com/ray-project/ray into work…

20cbf1f

…er_grpc

merge

85f2d73

raulchen added 2 commits July 11, 2019 22:57

fix

6600779

format

5e66995

noop

b60574d

stephanie-wang merged commit 1649f13 into ray-project:master Jul 11, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[direct call] changes raylet to push tasks to worker #5140

[direct call] changes raylet to push tasks to worker #5140

zhijunfu commented Jul 8, 2019

zhijunfu commented Jul 8, 2019

AmplabJenkins commented Jul 8, 2019

zhijunfu left a comment

AmplabJenkins commented Jul 8, 2019

AmplabJenkins commented Jul 8, 2019

AmplabJenkins commented Jul 9, 2019

raulchen commented Jul 9, 2019

AmplabJenkins commented Jul 9, 2019

AmplabJenkins commented Jul 9, 2019

stephanie-wang left a comment •

edited

Loading

zhijunfu commented Jul 10, 2019

AmplabJenkins commented Jul 10, 2019

AmplabJenkins commented Jul 10, 2019

zhijunfu commented Jul 10, 2019

AmplabJenkins commented Jul 10, 2019

AmplabJenkins commented Jul 10, 2019

AmplabJenkins commented Jul 10, 2019

AmplabJenkins commented Jul 10, 2019

AmplabJenkins commented Jul 10, 2019

AmplabJenkins commented Jul 10, 2019

AmplabJenkins commented Jul 11, 2019

AmplabJenkins commented Jul 11, 2019

AmplabJenkins commented Jul 11, 2019

AmplabJenkins commented Jul 11, 2019

stephanie-wang commented Jul 11, 2019

AmplabJenkins commented Jul 11, 2019

AmplabJenkins commented Jul 11, 2019

[direct call] changes raylet to push tasks to worker #5140

[direct call] changes raylet to push tasks to worker #5140

Conversation

zhijunfu commented Jul 8, 2019

What do these changes do?

Related issue number

Linter

zhijunfu commented Jul 8, 2019

AmplabJenkins commented Jul 8, 2019

zhijunfu left a comment

Choose a reason for hiding this comment

AmplabJenkins commented Jul 8, 2019

AmplabJenkins commented Jul 8, 2019

AmplabJenkins commented Jul 9, 2019

raulchen commented Jul 9, 2019

AmplabJenkins commented Jul 9, 2019

AmplabJenkins commented Jul 9, 2019

stephanie-wang left a comment • edited Loading

Choose a reason for hiding this comment

zhijunfu commented Jul 10, 2019

AmplabJenkins commented Jul 10, 2019

AmplabJenkins commented Jul 10, 2019

zhijunfu commented Jul 10, 2019

AmplabJenkins commented Jul 10, 2019

AmplabJenkins commented Jul 10, 2019

AmplabJenkins commented Jul 10, 2019

AmplabJenkins commented Jul 10, 2019

AmplabJenkins commented Jul 10, 2019

AmplabJenkins commented Jul 10, 2019

AmplabJenkins commented Jul 11, 2019

AmplabJenkins commented Jul 11, 2019

AmplabJenkins commented Jul 11, 2019

AmplabJenkins commented Jul 11, 2019

stephanie-wang commented Jul 11, 2019

AmplabJenkins commented Jul 11, 2019

AmplabJenkins commented Jul 11, 2019

stephanie-wang left a comment •

edited

Loading