[ID Refactor] Shorten the length of JobID to 4 bytes

[jovany-wang]

What do these changes do?

Some points in this PR:

(1) Shorten the length of JobID to 4 bytes.
(2) The job_id is generated by GCS for uniqueness.
(3) Embed job_id to driver id. The first 4 bytes of a driver id is the job_id bits, and rest 16bits must be filled with 0xFF.

Refer this doc for more details of ID Refactor.

Related issue number

Linter

• I've run scripts/format.sh to lint the changes in this PR.

[jovany-wang]

@zhijunfu cc

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15072/
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15094/
Test FAILed.

[jovany-wang]

@robertnishihara Does this commit make sense to you ?

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15106/
Test FAILed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15108/
Test PASSed.

[robertnishihara]

@jovany-wang why introduce worker.current_exported_id? Isn't that the same thing as the current job ID? Or maybe the IDs are different, but there is a one-to-one correspondence between them?

[jovany-wang]

@jovany-wang why introduce worker.current_exported_id? Isn't that the same thing as the current job ID? Or maybe the IDs are different, but there is a one-to-one correspondence between them?

@robertnishihara
Because this PR makes JobID incremented from 1 in a cluster instead of random one. that means we might get the same id in 2 sessions:

ray.init()  # job_id is 0x00000001
ray.shutdown()
ray.init()  # job_id is 0x00000001
ray.shutdown()

So, it's not appropriate to use job_id to check whether we have exported the classes and funcs in different sessions.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15112/
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1544/
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15236/
Test FAILed.

[pcmoritz]

Thanks for doing this, I left a few comments!

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1576/
Test FAILed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15266/
Test PASSed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15270/
Test PASSed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1580/
Test FAILed.

[jovany-wang]

@pcmoritz Do the responses make sense to you ?

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1600/
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1603/
Test FAILed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15290/
Test PASSed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15294/
Test PASSed.

[jovany-wang]

CI failures seems not related.
@pcmoritz Let's merge this ?

All comments were addressed. Let's merge to unblock others.