Update to latest arrow?

[alok]

The torch patch I added to arrow would fix #955, part of #1856, and part of #2036.

Is it feasible to update ray to use that version, or at least apply a patch in the build process to apply that one change?

[robertnishihara]

Yep, the relevant line is

ray/thirdparty/scripts/build_arrow.sh

Line 50 in c0e4c9d

git checkout 5f9cf9c96709f92e9ac4828cf3e106a165576ce7

.

It's just a matter of replacing that with a more recent commit (and updating the comment above).

Sometimes if there have been Arrow API changes, then other changes have to be made as well.

[richardliaw]

I tried git checkout df20683 (which I believe is master) and I wasn't able to compile, unfortunately.

[alok]

@richardliaw Can you try it with git checkout df9bdc10a9481ee328e22cf0dc20e17d4ca4f8d0? I would do it myself, but my installation is in a shaky place and I don't want to risk breaking it right now.

[richardliaw]

    Scanning dependencies of target ray_objlib
    [ 33%] Building CXX object src/ray/CMakeFiles/ray_objlib.dir/status.cc.o
    [ 34%] Building CXX object src/ray/CMakeFiles/ray_objlib.dir/id.cc.o
    [ 35%] Building CXX object src/ray/CMakeFiles/ray_objlib.dir/gcs/client.cc.o
    [ 36%] Building CXX object src/ray/CMakeFiles/ray_objlib.dir/gcs/tables.cc.o
    [ 36%] Building CXX object src/ray/CMakeFiles/ray_objlib.dir/gcs/task_table.cc.o
    [ 37%] Building CXX object src/ray/CMakeFiles/ray_objlib.dir/gcs/redis_context.cc.o
    [ 38%] Building CXX object src/ray/CMakeFiles/ray_objlib.dir/gcs/asio.cc.o
    [ 39%] Linking CXX static library libcommon.a
    [ 39%] Building CXX object src/ray/CMakeFiles/ray_objlib.dir/common/client_connection.cc.o
    [ 39%] Built target common
    [ 40%] Building CXX object src/ray/CMakeFiles/ray_objlib.dir/object_manager/object_manager_client_connection.cc.o
    [ 41%] Building CXX object src/ray/CMakeFiles/ray_objlib.dir/object_manager/connection_pool.cc.o
    [ 41%] Linking CXX static library liblocal_scheduler_client.a
    [ 41%] Built target local_scheduler_client
    [ 41%] Building CXX object src/ray/CMakeFiles/ray_objlib.dir/object_manager/object_buffer_pool.cc.o
    [ 42%] Building CXX object src/ray/CMakeFiles/ray_objlib.dir/object_manager/object_store_notification_manager.cc.o
    [ 43%] Building CXX object src/ray/CMakeFiles/ray_objlib.dir/object_manager/object_directory.cc.o
    [ 43%] Building CXX object src/ray/CMakeFiles/ray_objlib.dir/object_manager/object_manager.cc.o
    [ 44%] Building CXX object src/ray/CMakeFiles/ray_objlib.dir/raylet/monitor.cc.o
    [ 45%] Building CXX object src/ray/CMakeFiles/ray_objlib.dir/raylet/mock_gcs_client.cc.o
    [ 45%] Building CXX object src/ray/CMakeFiles/ray_objlib.dir/raylet/task.cc.o
    /Users/rliaw/Research/riselab/ray/src/ray/object_manager/object_store_notification_manager.cc:19:60: error: use of undeclared identifier 'kDeprecatedPlasmaDefaultReleaseDelay'; did you mean 'plasma::kDeprecatedPlasmaDefaultReleaseDelay'?
          store_client_.Connect(store_socket_name.c_str(), "", PLASMA_DEFAULT_RELEASE_DELAY));
                                                               ^
    /Users/rliaw/Research/riselab/ray/thirdparty/pkg/arrow/cpp/build/cpp-install/include/plasma/client.h:39:38: note: expanded from macro 'PLASMA_DEFAULT_RELEASE_DELAY'
    #define PLASMA_DEFAULT_RELEASE_DELAY kDeprecatedPlasmaDefaultReleaseDelay;
                                         ^
    /Users/rliaw/Research/riselab/ray/thirdparty/pkg/arrow/cpp/build/cpp-install/include/plasma/client.h:38:19: note: 'plasma::kDeprecatedPlasmaDefaultReleaseDelay' declared here
    constexpr int64_t kDeprecatedPlasmaDefaultReleaseDelay = 64;
                      ^
    /Users/rliaw/Research/riselab/ray/src/ray/object_manager/object_store_notification_manager.cc:19:60: error: 'kDeprecatedPlasmaDefaultReleaseDelay' is deprecated: PLASMA_DEFAULT_RELEASE_DELAY is deprecated [-Werror,-Wdeprecated-declarations]
          store_client_.Connect(store_socket_name.c_str(), "", PLASMA_DEFAULT_RELEASE_DELAY));
                                                               ^
    /Users/rliaw/Research/riselab/ray/thirdparty/pkg/arrow/cpp/build/cpp-install/include/plasma/client.h:39:38: note: expanded from macro 'PLASMA_DEFAULT_RELEASE_DELAY'
    #define PLASMA_DEFAULT_RELEASE_DELAY kDeprecatedPlasmaDefaultReleaseDelay;
                                         ^
    /Users/rliaw/Research/riselab/ray/thirdparty/pkg/arrow/cpp/build/cpp-install/include/plasma/client.h:37:1: note: 'kDeprecatedPlasmaDefaultReleaseDelay' has been explicitly marked deprecated here
    ARROW_DEPRECATED("PLASMA_DEFAULT_RELEASE_DELAY is deprecated")
    ^
    /Users/rliaw/Research/riselab/ray/thirdparty/pkg/arrow/cpp/build/cpp-install/include/arrow/util/macros.h:81:48: note: expanded from macro 'ARROW_DEPRECATED'
    #  define ARROW_DEPRECATED(...) __attribute__((deprecated(__VA_ARGS__)))
                                                   ^
    /Users/rliaw/Research/riselab/ray/src/ray/object_manager/object_store_notification_manager.cc:19:88: error: expected expression
          store_client_.Connect(store_socket_name.c_str(), "", PLASMA_DEFAULT_RELEASE_DELAY));
                                                                                           ^
    [ 46%] Building CXX object src/ray/CMakeFiles/ray_objlib.dir/raylet/task_execution_spec.cc.o
    3 errors generated.
    make[2]: *** [src/ray/CMakeFiles/ray_objlib.dir/object_manager/object_store_notification_manager.cc.o] Error 1
    make[2]: *** Waiting for unfinished jobs....
    make[1]: *** [src/ray/CMakeFiles/ray_objlib.dir/all] Error 2
    make: *** [all] Error 2
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/Users/rliaw/Research/riselab/ray/python/setup.py", line 141, in <module>
        license="Apache 2.0")
      File "/Users/rliaw/miniconda3/envs/ray/lib/python3.6/site-packages/setuptools/__init__.py", line 129, in setup
        return distutils.core.setup(**attrs)
      File "/Users/rliaw/miniconda3/envs/ray/lib/python3.6/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/Users/rliaw/miniconda3/envs/ray/lib/python3.6/distutils/dist.py", line 955, in run_commands
        self.run_command(cmd)
      File "/Users/rliaw/miniconda3/envs/ray/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/Users/rliaw/miniconda3/envs/ray/lib/python3.6/site-packages/setuptools/command/develop.py", line 36, in run
        self.install_for_development()
      File "/Users/rliaw/miniconda3/envs/ray/lib/python3.6/site-packages/setuptools/command/develop.py", line 136, in install_for_development
        self.run_command('build_ext')
      File "/Users/rliaw/miniconda3/envs/ray/lib/python3.6/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/Users/rliaw/miniconda3/envs/ray/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/Users/rliaw/Research/riselab/ray/python/setup.py", line 66, in run
        subprocess.check_call(["../build.sh", sys.executable])
      File "/Users/rliaw/miniconda3/envs/ray/lib/python3.6/subprocess.py", line 291, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['../build.sh', '/Users/rliaw/miniconda3/envs/ray/bin/python']' returned non-zero exit status 2.

    ----------------------------------------
  Rolling back uninstall of ray
Command "/Users/rliaw/miniconda3/envs/ray/bin/python -c "import setuptools, tokenize;__file__='/Users/rliaw/Research/riselab/ray/python/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps" failed with error code 1 in /Users/rliaw/Research/riselab/ray/python/

[pcmoritz]

Thanks, I will submit an arrow PR to fix that!

The fix is this:

In cpp/src/plasma/client.h, replace

#define PLASMA_DEFAULT_RELEASE_DELAY kDeprecatedPlasmaDefaultReleaseDelay;

with

#define PLASMA_DEFAULT_RELEASE_DELAY plasma::kDeprecatedPlasmaDefaultReleaseDelay

[pcmoritz]

Patch here, will merge when tests pass: apache/arrow#2063

[pschafhalter]

Updating Arrow should also resolve #1014 due to apache/arrow#2055.

[pcmoritz]

PR to update Ray: #2100

[pschafhalter]

Fixed by #2100.