Inline objects #3756

istoica · 2019-01-12T04:00:09Z

What do these changes do?

This pull request adds object data to the object entry in GCS, for small objects. This helps with performance optimization, and fault tolerance, as there is no need to reconstruct an object as long as its entry is in GCS.

AmplabJenkins · 2019-01-12T06:09:55Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/10782/
Test PASSed.

AmplabJenkins · 2019-01-12T22:18:48Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/10799/
Test PASSed.

AmplabJenkins · 2019-01-12T22:24:39Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/10797/
Test PASSed.

AmplabJenkins · 2019-01-12T22:25:13Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/10801/
Test PASSed.

AmplabJenkins · 2019-01-12T22:35:58Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/10800/
Test PASSed.

AmplabJenkins · 2019-01-13T18:15:38Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/10824/
Test PASSed.

pcmoritz · 2019-01-14T05:28:43Z

Travis is failing with

+/home/travis/build/ray-project/ray/build/src/ray/object_manager/object_manager_test /home/travis/build/ray-project/ray/python/ray/core/src/plasma/plasma_store_server
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from TestObjectManager
[ RUN      ] TestObjectManager.StartTestObjectManager
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0113 15:39:45.414286 16830 object_manager.cc:324] Invalid Push request ObjectID: dd20ed210f7ca5e66aa3b7b37e3999f7aee3e896 after waiting for 1000 ms.
	H������������&�XfU�9��+C���&
���s

Can you reproduce the failing of object_manager_test locally?

pcmoritz · 2019-01-14T05:30:51Z

maybe valgrind gives us some hint on what's going wrong here: https://travis-ci.com/ray-project/ray/jobs/170048915

pcmoritz · 2019-01-14T05:31:47Z

also some minor linting: https://travis-ci.com/ray-project/ray/jobs/170048914

pcmoritz · 2019-01-14T05:36:04Z

src/ray/object_manager/object_directory.cc

-    io_service_.post([callback, object_id, locations, has_been_created]() {
-      callback(object_id, locations, has_been_created);
+    bool inline_object_flag = it->second.inline_object_flag;
+    std::vector<uint8_t> inline_object_data = it->second.inline_object_data;


This will do a copy, it might be better to do const auto& inline_object_data = ...

src/ray/object_manager/object_directory.cc

src/ray/raylet/reconstruction_policy.cc

AmplabJenkins · 2019-01-14T08:26:35Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/10833/
Test PASSed.

AmplabJenkins · 2019-01-15T01:11:56Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/10860/
Test FAILed.

AmplabJenkins · 2019-01-30T01:12:45Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11275/
Test PASSed.

AmplabJenkins · 2019-01-30T09:19:25Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11292/
Test PASSed.

AmplabJenkins · 2019-01-30T20:05:53Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11305/
Test PASSed.

AmplabJenkins · 2019-01-31T02:29:46Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11323/
Test PASSed.

AmplabJenkins · 2019-01-31T07:23:13Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11348/
Test PASSed.

pcmoritz · 2019-01-31T22:11:33Z

test/runtest.py

+        def get(self):
+            return
+
+    def flush(actor):


The release delay is gone as of apache/arrow#3124, so we can get rid of this, yay!

Ah great, thanks!

Hmm actually, if I remove the flush call, this test sometimes fails on my laptop. Do you know why that is (not sure about the semantics of ray.internal.free for plasma)?

This is probably a race condition (ray.internal.free is using the plasma client in the raylet which is different from the one in the worker that get is called on).

One way around that is to use ray.workers.global_worker.plasma_client.delete, which uses the same plasma client that will be used for the get, so that should work.

Hmm this still seems to fail occasionally with plasma_client.delete (maybe 1/100 times). Any idea why or do you think it's a bug somewhere else?

pcmoritz · 2019-01-31T22:21:42Z

src/ray/object_manager/object_manager.cc

-            if (!client_ids.empty()) {
+                          const std::unordered_set<ClientID> &client_ids,
+                          bool inline_object_flag,
+                          const std::vector<uint8_t> inline_object_data,


why not const std::vector<uint8_t>& inline_object_data here?

Oops, thank you!

pcmoritz · 2019-01-31T22:22:50Z

src/ray/object_manager/object_manager.cc

+    // Inline object is not in the local object store. Create it from
+    // inline_object_data, and inline_object_metadata, respectively.
+    //
+    // Since this function is called on notification or when reading the


AmplabJenkins · 2019-02-01T01:36:35Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11362/
Test PASSed.

AmplabJenkins · 2019-02-01T02:42:01Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11369/
Test PASSed.

AmplabJenkins · 2019-02-02T02:09:10Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11420/
Test PASSed.

AmplabJenkins · 2019-02-05T22:46:39Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11581/
Test FAILed.

This reverts commit f987572.

* Revert "Inline objects (#3756)" This reverts commit f987572. * fix rebase problems * more rebase fixes * add back debug statement

istoica added 6 commits January 9, 2019 15:48

added store_client_ to object_manager and node_manager

71da1ed

half through...

e43a221

all code in, and compiling! Nothing tested though...

94cea3a

something is working ;-)

bd71d14

added a few more comments

2c058f5

now, add only one entry to the in GCS for inlined objects

ded447e

istoica requested a review from stephanie-wang January 12, 2019 04:00

istoica and others added 5 commits January 12, 2019 11:37

more comments

3e6f9b2

remove a spurious todo

4855736

some comment updates

71bdf21

add test

52e2611

Merge branch 'inline-objects' into inline-objects

9931dd1

added support for meta data for inline objects

88ed5e2

pcmoritz reviewed Jan 14, 2019

View reviewed changes

src/ray/object_manager/object_directory.cc Outdated Show resolved Hide resolved

pcmoritz reviewed Jan 14, 2019

View reviewed changes

src/ray/raylet/reconstruction_policy.cc Outdated Show resolved Hide resolved

pcmoritz reviewed Jan 14, 2019

View reviewed changes

src/ray/raylet/reconstruction_policy.cc Outdated Show resolved Hide resolved

avoid some copies

4b6dd43

Initialize plasma client in tests

0869d26

stephanie-wang changed the base branch from inline-objects to master January 30, 2019 06:21

fix build

f14d1f4

stephanie-wang added 2 commits January 30, 2019 09:22

Merge branch 'master' into inline-objects

feb1e68

Fix build

ffcd402

Merge branch 'master' into inline-objects

90721a6

lint

f32a7b0

pcmoritz reviewed Jan 31, 2019

View reviewed changes

stephanie-wang added 2 commits January 31, 2019 14:52

Use const ref

d284fa8

Fixes, don't let object manager hang

a374ce6

stephanie-wang added 2 commits February 1, 2019 13:54

Increase object transfer retry time for travis?

40b5d27

Fix test

b663753

stephanie-wang added 2 commits February 4, 2019 16:26

Fix test?

c29ab7c

Add internal config to java, fix PlasmaFreeTest

08f705a

stephanie-wang merged commit f987572 into ray-project:master Feb 7, 2019

robertnishihara mentioned this pull request Feb 7, 2019

Valgrind object manager test failure. #3979

Closed

guoyuhong mentioned this pull request Feb 10, 2019

Fix CI test failures #4007

Merged

stephanie-wang mentioned this pull request Feb 12, 2019

Increase timeout for object manager valgrind tests #4027

Merged

pcmoritz added a commit to pcmoritz/ray-1 that referenced this pull request Feb 22, 2019

Revert "Inline objects (ray-project#3756)"

eb5a9ad

This reverts commit f987572.

stephanie-wang pushed a commit that referenced this pull request Feb 23, 2019

Revert inline objects PR (#4125)

9b3ce3e

* Revert "Inline objects (#3756)" This reverts commit f987572. * fix rebase problems * more rebase fixes * add back debug statement

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Inline objects #3756

Inline objects #3756

istoica commented Jan 12, 2019 •

edited

Loading

AmplabJenkins commented Jan 12, 2019

AmplabJenkins commented Jan 12, 2019

AmplabJenkins commented Jan 12, 2019

AmplabJenkins commented Jan 12, 2019

AmplabJenkins commented Jan 12, 2019

AmplabJenkins commented Jan 13, 2019

pcmoritz commented Jan 14, 2019

pcmoritz commented Jan 14, 2019

pcmoritz commented Jan 14, 2019

pcmoritz Jan 14, 2019

AmplabJenkins commented Jan 14, 2019

AmplabJenkins commented Jan 15, 2019

AmplabJenkins commented Jan 30, 2019

AmplabJenkins commented Jan 30, 2019

AmplabJenkins commented Jan 30, 2019

AmplabJenkins commented Jan 31, 2019

AmplabJenkins commented Jan 31, 2019

pcmoritz Jan 31, 2019

stephanie-wang Jan 31, 2019

stephanie-wang Jan 31, 2019

pcmoritz Jan 31, 2019 •

edited

Loading

stephanie-wang Feb 1, 2019

pcmoritz Jan 31, 2019

stephanie-wang Jan 31, 2019

pcmoritz Jan 31, 2019

AmplabJenkins commented Feb 1, 2019

AmplabJenkins commented Feb 1, 2019

AmplabJenkins commented Feb 2, 2019

AmplabJenkins commented Feb 5, 2019

Inline objects #3756

Inline objects #3756

Conversation

istoica commented Jan 12, 2019 • edited Loading

What do these changes do?

AmplabJenkins commented Jan 12, 2019

AmplabJenkins commented Jan 12, 2019

AmplabJenkins commented Jan 12, 2019

AmplabJenkins commented Jan 12, 2019

AmplabJenkins commented Jan 12, 2019

AmplabJenkins commented Jan 13, 2019

pcmoritz commented Jan 14, 2019

pcmoritz commented Jan 14, 2019

pcmoritz commented Jan 14, 2019

Choose a reason for hiding this comment

AmplabJenkins commented Jan 14, 2019

AmplabJenkins commented Jan 15, 2019

AmplabJenkins commented Jan 30, 2019

AmplabJenkins commented Jan 30, 2019

AmplabJenkins commented Jan 30, 2019

AmplabJenkins commented Jan 31, 2019

AmplabJenkins commented Jan 31, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pcmoritz Jan 31, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AmplabJenkins commented Feb 1, 2019

AmplabJenkins commented Feb 1, 2019

AmplabJenkins commented Feb 2, 2019

AmplabJenkins commented Feb 5, 2019

istoica commented Jan 12, 2019 •

edited

Loading

pcmoritz Jan 31, 2019 •

edited

Loading