Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update arrow to latest master #2100

Merged
merged 17 commits into from
May 24, 2018
Merged

Update arrow to latest master #2100

merged 17 commits into from
May 24, 2018

Conversation

pcmoritz
Copy link
Contributor

This includes the following changes:

Serialization fix for Python 2 integers: apache/arrow#2055
Plasma performance improvements: apache/arrow#2056, apache/arrow#2015, apache/arrow#2022
PyTorch 0.4 Tensor serialization fix: apache/arrow#2033

@pcmoritz pcmoritz mentioned this pull request May 19, 2018
Closed
@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5492/
Test PASSed.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5504/
Test PASSed.

@robertnishihara
Copy link
Collaborator

Looks like there are compilation errors on Mac

[ 44%] �[32mBuilding CXX object src/ray/CMakeFiles/ray_objlib.dir/object_manager/object_directory.cc.o�[0m

�[1m/Users/travis/build/robertnishihara/ray-private-travis/src/ray/object_manager/object_store_notification_manager.cc:19:60: �[0m�[0;1;31merror: �[0m�[1m'kDeprecatedPlasmaDefaultReleaseDelay' is deprecated: PLASMA_DEFAULT_RELEASE_DELAY is deprecated [-Werror,-Wdeprecated-declarations]�[0m

      store_client_.Connect(store_socket_name.c_str(), "", PLASMA_DEFAULT_RELEASE_DELAY));

�[0;1;32m                                                           ^

�[0m�[1m/Users/travis/build/robertnishihara/ray-private-travis/thirdparty/pkg/arrow/cpp/build/cpp-install/include/plasma/client.h:39:46: �[0m�[0;1;30mnote: �[0mexpanded from macro 'PLASMA_DEFAULT_RELEASE_DELAY'�[0m

#define PLASMA_DEFAULT_RELEASE_DELAY plasma::kDeprecatedPlasmaDefaultReleaseDelay

�[0;1;32m                                             ^

�[0m�[1m/Users/travis/build/robertnishihara/ray-private-travis/thirdparty/pkg/arrow/cpp/build/cpp-install/include/plasma/client.h:38:19: �[0m�[0;1;30mnote: �[0m'kDeprecatedPlasmaDefaultReleaseDelay' has been explicitly marked deprecated here�[0m

constexpr int64_t kDeprecatedPlasmaDefaultReleaseDelay = 64;

�[0;1;32m                  ^

�[0m1 error generated.

@robertnishihara
Copy link
Collaborator

Also, this looks related (it happened a couple times on private travis).

======================================================================
FAIL: testLocalSchedulerFailed (__main__.ComponentFailureTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/component_failures_test.py", line 200, in testLocalSchedulerFailed
    True)
  File "test/component_failures_test.py", line 182, in check_components_alive
    self.assertTrue(component.poll() is None)
AssertionError: False is not true

----------------------------------------------------------------------
Ran 9 tests in 62.892s

FAILED (failures=1)

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5519/
Test FAILed.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5518/
Test PASSed.

@robertnishihara
Copy link
Collaborator

There's still a linting error

ObjectStoreNotificationManager::ObjectStoreNotificationManager(
     boost::asio::io_service &io_service, const std::string &store_socket_name)
     : store_client_(), socket_(io_service) {
-  ARROW_CHECK_OK(
-      store_client_.Connect(store_socket_name.c_str(), "", plasma::kPlasmaDefaultReleaseDelay));
+  ARROW_CHECK_OK(store_client_.Connect(store_socket_name.c_str(), "",
+                                       plasma::kPlasmaDefaultReleaseDelay));

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5522/
Test PASSed.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5528/
Test PASSed.

@robertnishihara
Copy link
Collaborator

Unfortunately this is still happening on Linux

======================================================================
FAIL: testLocalSchedulerFailed (__main__.ComponentFailureTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/component_failures_test.py", line 200, in testLocalSchedulerFailed
    True)
  File "test/component_failures_test.py", line 182, in check_components_alive
    self.assertTrue(component.poll() is None)
AssertionError: False is not true

----------------------------------------------------------------------
Ran 9 tests in 62.430s

FAILED (failures=1)

@pcmoritz
Copy link
Contributor Author

The test failure is caused by this segfault I think:

Program received signal SIGSEGV, Segmentation fault.
std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> >::unique_ptr(std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> >&&) (
    __u=<optimized out>, this=0x800000000) at /usr/include/c++/5/bits/unique_ptr.h:464
464	      : _M_t(__u.release(), std::forward<deleter_type>(__u.get_deleter())) { }
(gdb) bt
#0  std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> >::unique_ptr(std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> >&&) (
    __u=<optimized out>, this=0x800000000) at /usr/include/c++/5/bits/unique_ptr.h:464
#1  __gnu_cxx::new_allocator<std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> > >::construct<std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> >, std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> > >(std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> >*, std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> >&&) (__p=0x800000000, this=0x2636cc0) at /usr/include/c++/5/ext/new_allocator.h:120
#2  std::allocator_traits<std::allocator<std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> > > >::construct<std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> >, std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> > >(std::allocator<std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> > >&, std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> >*, std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> >&&) (
    __p=0x800000000, __a=...) at /usr/include/c++/5/bits/alloc_traits.h:530
#3  std::deque<std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> >, std::allocator<std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> > > >::emplace_back<std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> > >(std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> >&&) (
    this=0x2636cc0) at /usr/include/c++/5/bits/deque.tcc:155
#4  plasma::PlasmaStore::push_notification (this=this@entry=0x2631d60, object_info=object_info@entry=0x2638f78)
    at /home/ubuntu/ray/thirdparty/build/arrow/cpp/src/plasma/store.cc:633
#5  0x000000000041124f in plasma::PlasmaStore::seal_object (this=this@entry=0x2631d60, object_id=..., 
    digest=digest@entry=0x7ffc5d1d39a0 "\001\253,\271cT", <incomplete sequence \352>) at /home/ubuntu/ray/thirdparty/build/arrow/cpp/src/plasma/store.cc:439
#6  0x0000000000411a34 in plasma::PlasmaStore::process_message (this=0x2631d60, client=<optimized out>) at /home/ubuntu/ray/thirdparty/build/arrow/cpp/src/plasma/store.cc:727
#7  0x0000000000413979 in plasma::PlasmaStore::<lambda(int)>::operator() (events=<optimized out>, __closure=<optimized out>)
    at /home/ubuntu/ray/thirdparty/build/arrow/cpp/src/plasma/store.cc:524
#8  std::_Function_handler<void(int), plasma::PlasmaStore::connect_client(int)::<lambda(int)> >::_M_invoke(const std::_Any_data &, <unknown type in /home/ubuntu/ray/python/ray/core/src/plasma/plasma_store, CU 0x0, DIE 0x82032>) (__functor=..., __args#0=<optimized out>) at /usr/include/c++/5/functional:1871
#9  0x00000000004181da in std::function<void (int)>::operator()(int) const (__args#0=1, this=<optimized out>) at /usr/include/c++/5/functional:2267
#10 plasma::EventLoop::FileEventCallback (loop=<optimized out>, fd=<optimized out>, context=<optimized out>, events=<optimized out>)
    at /home/ubuntu/ray/thirdparty/build/arrow/cpp/src/plasma/events.cc:28
#11 0x0000000000433eb0 in aeProcessEvents (eventLoop=eventLoop@entry=0x2624cc0, flags=flags@entry=3)
    at /home/ubuntu/ray/thirdparty/build/arrow/cpp/src/plasma/thirdparty/ae/ae.c:412
#12 0x00000000004342cb in aeMain (eventLoop=0x2624cc0) at /home/ubuntu/ray/thirdparty/build/arrow/cpp/src/plasma/thirdparty/ae/ae.c:455
#13 0x0000000000418388 in plasma::EventLoop::Start (this=<optimized out>) at /home/ubuntu/ray/thirdparty/build/arrow/cpp/src/plasma/events.cc:69
#14 0x000000000040d2a5 in plasma::PlasmaStoreRunner::Start (use_one_memory_mapped_file=false, hugepages_enabled=false, directory="P;\035]", system_memory=<optimized out>, 
    socket_name=0x7ffc5d1d51d4 "/tmp/plasma_store66283217", this=0x2624c20) at /home/ubuntu/ray/thirdparty/build/arrow/cpp/src/plasma/store.cc:785
#15 plasma::start_server (socket_name=socket_name@entry=0x7ffc5d1d51d4 "/tmp/plasma_store66283217", system_memory=<optimized out>, plasma_directory="/dev/shm", 
    hugepages_enabled=hugepages_enabled@entry=false, use_one_memory_mapped_file=use_one_memory_mapped_file@entry=false)
    at /home/ubuntu/ray/thirdparty/build/arrow/cpp/src/plasma/store.cc:820

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5559/
Test FAILed.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5557/
Test PASSed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5564/
Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5563/
Test FAILed.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5561/
Test PASSed.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5562/
Test PASSed.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5567/
Test PASSed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5570/
Test FAILed.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5586/
Test PASSed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5588/
Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5589/
Test FAILed.

@pcmoritz
Copy link
Contributor Author

Jenkins retest this please

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5605/
Test PASSed.

@richardliaw
Copy link
Contributor

@pcmoritz tests look like they're passing?

@robertnishihara robertnishihara merged commit 225608e into ray-project:master May 24, 2018
@robertnishihara robertnishihara deleted the update-arrow branch May 24, 2018 07:26
@pschafhalter pschafhalter mentioned this pull request May 24, 2018
Closed
alok added a commit to alok/ray that referenced this pull request May 25, 2018
@alok
Merge branch 'master' into fix-a3c-torch
e865a09 
* master:
  Prototype named actors. (ray-project#2129)
  Update arrow to latest master (ray-project#2100)
  [DataFrame] Speed up dtypes (ray-project#2118)
  do not fetch from dead Plasma Manager (ray-project#2116)
  [DataFrame] Refactor GroupBy Methods and Implement Reindex (ray-project#2101)
  Initial Support for Airspeed Velocity (ray-project#2113)
  Use automatic memory management in Redis modules. (ray-project#1797)
  [DataFrame] Test bugfixes (ray-project#2111)
  [DataFrame] Update initializations of IndexMetadata which use outdated APIs (ray-project#2103)
This was referenced May 25, 2018
Closed
Merged
alok added a commit to alok/ray that referenced this pull request May 30, 2018
@alok
Merge branch 'fix-a3c-torch' into trpo
a62fa6e 
* fix-a3c-torch:
  Prototype named actors. (ray-project#2129)
  Update arrow to latest master (ray-project#2100)
  [DataFrame] Speed up dtypes (ray-project#2118)
  do not fetch from dead Plasma Manager (ray-project#2116)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@robertnishihara robertnishihara robertnishihara approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@pcmoritz @AmplabJenkins @robertnishihara @richardliaw