Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ray doesn't build on Ubuntu 19.10 due to GRPC #5984

Closed
mawright opened this issue Oct 23, 2019 · 8 comments · Fixed by #6002
Closed

Ray doesn't build on Ubuntu 19.10 due to GRPC #5984

mawright opened this issue Oct 23, 2019 · 8 comments · Fixed by #6002

Comments

@mawright
Copy link
Contributor

System information

  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 19.10
  • Ray installed from (source or binary): Source
  • Ray version: 34fbc7f
  • Python version: Anaconda 3.7
  • Exact command to reproduce: pip install -e . --verbose

Describe the problem

Source code / logs

pip install -e . --verbose
Created temporary directory: /tmp/pip-ephem-wheel-cache-ri_9ne9g
Created temporary directory: /tmp/pip-req-tracker-03xdp_qj
Created requirements tracker '/tmp/pip-req-tracker-03xdp_qj'
Created temporary directory: /tmp/pip-install-czi9468r
Obtaining file:///home/matt/Code/ray/python
  Added file:///home/matt/Code/ray/python to build tracker '/tmp/pip-req-tracker-03xdp_qj'
    Running setup.py (path:/home/matt/Code/ray/python/setup.py) egg_info for package from file:///home/matt/Code/ray/python
    Running command python setup.py egg_info
    running egg_info
    writing ray.egg-info/PKG-INFO
    writing dependency_links to ray.egg-info/dependency_links.txt
    writing entry points to ray.egg-info/entry_points.txt
    writing requirements to ray.egg-info/requires.txt
    writing top-level names to ray.egg-info/top_level.txt
    reading manifest file 'ray.egg-info/SOURCES.txt'
    writing manifest file 'ray.egg-info/SOURCES.txt'
  Source in /home/matt/Code/ray/python has version 0.8.0.dev6, which satisfies requirement ray==0.8.0.dev6 from file:///home/matt/Code/ray/python
  Removed ray==0.8.0.dev6 from file:///home/matt/Code/ray/python from build tracker '/tmp/pip-req-tracker-03xdp_qj'
Requirement already satisfied: numpy>=1.14 in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from ray==0.8.0.dev6) (1.17.2)
Requirement already satisfied: filelock in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from ray==0.8.0.dev6) (3.0.12)
Requirement already satisfied: jsonschema in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from ray==0.8.0.dev6) (3.1.1)
Requirement already satisfied: funcsigs in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from ray==0.8.0.dev6) (1.0.2)
Requirement already satisfied: click in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from ray==0.8.0.dev6) (7.0)
Requirement already satisfied: colorama in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from ray==0.8.0.dev6) (0.4.1)
Requirement already satisfied: pytest in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from ray==0.8.0.dev6) (5.2.1)
Requirement already satisfied: pyyaml in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from ray==0.8.0.dev6) (5.1.2)
Requirement already satisfied: redis>=3.3.2 in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from ray==0.8.0.dev6) (3.3.8)
Requirement already satisfied: six>=1.0.0 in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from ray==0.8.0.dev6) (1.12.0)
Requirement already satisfied: protobuf>=3.8.0 in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from ray==0.8.0.dev6) (3.9.2)
Requirement already satisfied: attrs>=17.4.0 in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from jsonschema->ray==0.8.0.dev6) (19.3.0)
Requirement already satisfied: setuptools in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from jsonschema->ray==0.8.0.dev6) (41.4.0)
Requirement already satisfied: importlib-metadata in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from jsonschema->ray==0.8.0.dev6) (0.23)
Requirement already satisfied: pyrsistent>=0.14.0 in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from jsonschema->ray==0.8.0.dev6) (0.15.4)
Requirement already satisfied: py>=1.5.0 in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from pytest->ray==0.8.0.dev6) (1.8.0)
Requirement already satisfied: packaging in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from pytest->ray==0.8.0.dev6) (19.2)
Requirement already satisfied: more-itertools>=4.0.0 in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from pytest->ray==0.8.0.dev6) (7.2.0)
Requirement already satisfied: atomicwrites>=1.0 in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from pytest->ray==0.8.0.dev6) (1.3.0)
Requirement already satisfied: pluggy<1.0,>=0.12 in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from pytest->ray==0.8.0.dev6) (0.13.0)
Requirement already satisfied: wcwidth in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from pytest->ray==0.8.0.dev6) (0.1.7)
Requirement already satisfied: zipp>=0.5 in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from importlib-metadata->jsonschema->ray==0.8.0.dev6) (0.6.0)
Requirement already satisfied: pyparsing>=2.0.2 in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from packaging->pytest->ray==0.8.0.dev6) (2.4.2)
Installing collected packages: ray
  Running setup.py develop for ray
    Running command /home/matt/anaconda3/envs/gpu/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/matt/Code/ray/python/setup.py'"'"'; __file__='"'"'/home/matt/Code/ray/python/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps
    running develop
    running egg_info
    writing ray.egg-info/PKG-INFO
    writing dependency_links to ray.egg-info/dependency_links.txt
    writing entry points to ray.egg-info/entry_points.txt
    writing requirements to ray.egg-info/requires.txt
    writing top-level names to ray.egg-info/top_level.txt
    reading manifest file 'ray.egg-info/SOURCES.txt'
    writing manifest file 'ray.egg-info/SOURCES.txt'
    running build_ext
    + set -e
    +++ dirname ../build.sh
    ++ cd ..
    ++ pwd
    + ROOT_DIR=/home/matt/Code/ray
    ++ uname
    + unamestr=Linux
    + [[ Linux == \L\i\n\u\x ]]
    + PARALLEL=1
    + RAY_BUILD_PYTHON=YES
    + RAY_BUILD_JAVA=NO
    + PYTHON_EXECUTABLE=
    + BUILD_DIR=
    + [[ 2 -gt 0 ]]
    + key=-p
    + case $key in
    + PYTHON_EXECUTABLE=/home/matt/anaconda3/envs/gpu/bin/python
    + shift
    + shift
    + [[ 0 -gt 0 ]]
    + [[ -z /home/matt/anaconda3/envs/gpu/bin/python ]]
    + echo 'Using Python executable /home/matt/anaconda3/envs/gpu/bin/python.'
    Using Python executable /home/matt/anaconda3/envs/gpu/bin/python.
    ++ PATH=/home/matt/anaconda3/envs/gpu/bin:/home/matt/anaconda3/condabin:/home/matt/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/matt/.bazel/bin
    ++ which bazel
    + BAZEL_EXECUTABLE=/home/matt/bin/bazel
    + echo 'Using Bazel executable /home/matt/bin/bazel.'
    Using Bazel executable /home/matt/bin/bazel.
    + RAY_BUILD_PYTHON=YES
    + RAY_BUILD_JAVA=NO
    + bash /home/matt/Code/ray/setup_thirdparty.sh /home/matt/anaconda3/envs/gpu/bin/python
    + set -e
    +++ dirname /home/matt/Code/ray/setup_thirdparty.sh
    ++ cd /home/matt/Code/ray
    ++ pwd
    + ROOT_DIR=/home/matt/Code/ray
    + [[ -z /home/matt/anaconda3/envs/gpu/bin/python ]]
    + PYTHON_EXECUTABLE=/home/matt/anaconda3/envs/gpu/bin/python
    + echo 'Using Python executable /home/matt/anaconda3/envs/gpu/bin/python.'
    Using Python executable /home/matt/anaconda3/envs/gpu/bin/python.
    + RAY_BUILD_PYTHON=YES
    + RAY_BUILD_JAVA=NO
    + /home/matt/Code/ray/thirdparty/scripts/setup.sh /home/matt/anaconda3/envs/gpu/bin/python
    + set -e
    +++ dirname /home/matt/Code/ray/thirdparty/scripts/setup.sh
    ++ cd /home/matt/Code/ray/thirdparty/scripts
    ++ pwd
    + TP_SCRIPT_DIR=/home/matt/Code/ray/thirdparty/scripts
    + TP_DIR=/home/matt/Code/ray/thirdparty/scripts/..
    + mkdir -p /home/matt/Code/ray/thirdparty/scripts/../build
    + mkdir -p /home/matt/Code/ray/thirdparty/scripts/../pkg
    + [[ -z /home/matt/anaconda3/envs/gpu/bin/python ]]
    + PYTHON_EXECUTABLE=/home/matt/anaconda3/envs/gpu/bin/python
    + echo 'Using Python executable /home/matt/anaconda3/envs/gpu/bin/python.'
    Using Python executable /home/matt/anaconda3/envs/gpu/bin/python.
    + [[ NO == \Y\E\S ]]
    + [[ YES == \Y\E\S ]]
    + echo 'Python library will be built.'
    Python library will be built.
    ++ uname
    + unamestr=Linux
    + BUILD_DIR=/home/matt/Code/ray/build/
    + '[' '!' -d /home/matt/Code/ray/build/ ']'
    + pushd /home/matt/Code/ray/build/
    ~/Code/ray/build ~/Code/ray/python
    + '[' -z '' ']'
    + /home/matt/anaconda3/envs/gpu/bin/python -m pip install -q --target=/home/matt/Code/ray/python/ray/pyarrow_files pyarrow==0.14.0.RAY --find-links https://s3-us-west-2.amazonaws.com/arrow-wheels/516e15028091b5e287200b5df77d77f72d9a6c9a/index.html
    WARNING: Target directory /home/matt/Code/ray/python/ray/pyarrow_files/pyarrow already exists. Specify --upgrade to force replacement.
    WARNING: Target directory /home/matt/Code/ray/python/ray/pyarrow_files/numpy-1.17.3.dist-info already exists. Specify --upgrade to force replacement.
    WARNING: Target directory /home/matt/Code/ray/python/ray/pyarrow_files/six-1.12.0.dist-info already exists. Specify --upgrade to force replacement.
    WARNING: Target directory /home/matt/Code/ray/python/ray/pyarrow_files/six.py already exists. Specify --upgrade to force replacement.
    WARNING: Target directory /home/matt/Code/ray/python/ray/pyarrow_files/pyarrow-0.14.0.RAY.dist-info already exists. Specify --upgrade to force replacement.
    WARNING: Target directory /home/matt/Code/ray/python/ray/pyarrow_files/__pycache__ already exists. Specify --upgrade to force replacement.
    WARNING: Target directory /home/matt/Code/ray/python/ray/pyarrow_files/numpy already exists. Specify --upgrade to force replacement.
    WARNING: Target directory /home/matt/Code/ray/python/ray/pyarrow_files/bin already exists. Specify --upgrade to force replacement.
    + export PYTHON_BIN_PATH=/home/matt/anaconda3/envs/gpu/bin/python
    + PYTHON_BIN_PATH=/home/matt/anaconda3/envs/gpu/bin/python
    + '[' NO == YES ']'
    + '[' YES == YES ']'
    + /home/matt/bin/bazel build //:ray_pkg --verbose_failures
    Extracting Bazel installation...
    Starting local Bazel server and connecting to it...
    Loading:
    Loading: 0 packages loaded
    DEBUG: Rule 'com_github_checkstyle_java' indicated that a canonical reproducible form can be obtained by modifying arguments shallow_since = "1552542575 +0800"
    Loading: 0 packages loaded
    DEBUG: Rule 'com_github_nelhage_rules_boost' indicated that a canonical reproducible form can be obtained by modifying arguments shallow_since = "1556014830 +0800"
    INFO: SHA256 (https://github.com/jovany-wang/prometheus-cpp/archive/master.zip) = d0c773da8af3db99c543dd0413f4427d835170eddfd517bfeba104236a8d2c07
    DEBUG: Rule 'com_github_jupp0r_prometheus_cpp' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "d0c773da8af3db99c543dd0413f4427d835170eddfd517bfeba104236a8d2c07"
    Loading: 0 packages loaded
    Loading: 0 packages loaded
    Loading: 0 packages loaded
    Loading: 0 packages loaded
    Loading: 0 packages loaded
    Loading: 0 packages loaded
    INFO: SHA256 (https://github.com/grpc/grpc/archive/76a381869413834692b8ed305fbe923c0f9c4472.tar.gz) = b5efbe086b9a00826a3f830094312e6d1647157b5a5e7954a8ac4179bce3de8b
    Loading: 0 packages loaded
    DEBUG: Rule 'com_github_grpc_grpc' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "b5efbe086b9a00826a3f830094312e6d1647157b5a5e7954a8ac4179bce3de8b"
    INFO: SHA256 (https://github.com/google/bazel-common/archive/f1115e0f777f08c3cdb115526c4e663005bec69b.zip) = 1e05a4791cc3470d3ecf7edb556f796b1d340359f1c4d293f175d4d0946cf84c
    DEBUG: Rule 'bazel_common' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "1e05a4791cc3470d3ecf7edb556f796b1d340359f1c4d293f175d4d0946cf84c"
    INFO: SHA256 (https://github.com/bazelbuild/bazel-skylib/archive/0.6.0.tar.gz) = eb5c57e4c12e68c0c20bc774bfbc60a568e800d025557bc4ea022c6479acc867
    DEBUG: Rule 'bazel_skylib' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "eb5c57e4c12e68c0c20bc774bfbc60a568e800d025557bc4ea022c6479acc867"
    Loading: 0 packages loaded
        currently loading:
    Loading: 0 packages loaded
        currently loading:
    DEBUG: Rule 'com_github_google_flatbuffers' indicated that a canonical reproducible form can be obtained by modifying arguments shallow_since = "1547755012 -0800"
    Analyzing: target //:ray_pkg (1 packages loaded, 0 targets configured)
    Analyzing: target //:ray_pkg (3 packages loaded, 16 targets configured)
    Analyzing: target //:ray_pkg (3 packages loaded, 16 targets configured)
    DEBUG: Rule 'plasma' indicated that a canonical reproducible form can be obtained by modifying arguments shallow_since = "1570611764 -0700"
    INFO: SHA256 (https://github.com/census-instrumentation/opencensus-cpp/archive/3aa11f20dd610cb8d2f7c62e58d1e69196aadf11.zip) = 92eef77c44d01e8472f68a2f1329919a1bb59317a4bb1e4d76081ab5c13a56d6
    DEBUG: Rule 'io_opencensus_cpp' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "92eef77c44d01e8472f68a2f1329919a1bb59317a4bb1e4d76081ab5c13a56d6"
    DEBUG: Rule 'com_github_google_glog' indicated that a canonical reproducible form can be obtained by modifying arguments shallow_since = "1550458164 +0900"
    DEBUG: Rule 'com_github_gflags_gflags' indicated that a canonical reproducible form can be obtained by modifying arguments commit = "e171aa2d15ed9eb17054558e0b3a6a413bb01067", shallow_since = "1541971260 +0000" and dropping ["tag"]
    DEBUG: Rule 'com_google_absl' indicated that a canonical reproducible form can be obtained by modifying arguments shallow_since = "1565288385 -0400"
    Analyzing: target //:ray_pkg (29 packages loaded, 486 targets configured)
    DEBUG: Rule 'com_google_googletest' indicated that a canonical reproducible form can be obtained by modifying arguments shallow_since = "1534270723 -0700"
    Analyzing: target //:ray_pkg (48 packages loaded, 6524 targets configured)
    INFO: SHA256 (https://boringssl.googlesource.com/boringssl/+archive/afc30d43eef92979b05776ec0963c9cede5fb80f.tar.gz) = 939ea34bd847d8ef5dcabed747e2a65c5d8e82e6b63758c7d1fb11b806b495e7
    DEBUG: Rule 'boringssl' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "939ea34bd847d8ef5dcabed747e2a65c5d8e82e6b63758c7d1fb11b806b495e7"
    Analyzing: target //:ray_pkg (49 packages loaded, 7007 targets configured)
    Analyzing: target //:ray_pkg (50 packages loaded, 7031 targets configured)
    Analyzing: target //:ray_pkg (50 packages loaded, 7031 targets configured)
    Analyzing: target //:ray_pkg (50 packages loaded, 7031 targets configured)
    DEBUG: Rule 'cython' indicated that a canonical reproducible form can be obtained by modifying arguments shallow_since = "1547888711 +0100"
    INFO: Analyzed target //:ray_pkg (51 packages loaded, 7305 targets configured).
    INFO: Found 1 target...
    [0 / 14] [Prepa] BazelWorkspaceStatusAction stable-status.txt
    [134 / 335] [Analy] Compiling external/com_google_protobuf/src/google/protobuf/compiler/java/java_context.cc [for host]; 15s ... (145 actions, 10 running)
    [208 / 335] [Analy] Compiling external/com_google_protobuf/src/google/protobuf/compiler/java/java_helpers.cc [for host]; 34s ... (71 actions, 11 running)
    INFO: From Compiling external/com_google_protobuf/src/google/protobuf/descriptor.cc [for host]:
    external/com_google_protobuf/src/google/protobuf/descriptor.cc: In member function 'google::protobuf::Symbol google::protobuf::DescriptorPool::NewPlaceholderWithMutexHeld(const string&, google::protobuf::DescriptorPool::PlaceholderType) const':
    external/com_google_protobuf/src/google/protobuf/descriptor.cc:3901:58: warning: 'void* memset(void*, int, size_t)' clearing an object of type 'class google::protobuf::EnumDescriptor' with no trivial copy-assignment; use assignment or value-initialization instead [-Wclass-memaccess]
     3901 |     memset(placeholder_enum, 0, sizeof(*placeholder_enum));
          |                                                          ^
    In file included from external/com_google_protobuf/src/google/protobuf/message.h:121,
                     from external/com_google_protobuf/src/google/protobuf/descriptor.pb.h:30,
                     from external/com_google_protobuf/src/google/protobuf/descriptor.cc:50:
    external/com_google_protobuf/src/google/protobuf/descriptor.h:904:23: note: 'class google::protobuf::EnumDescriptor' declared here
      904 | class PROTOBUF_EXPORT EnumDescriptor {
          |                       ^~~~~~~~~~~~~~
    external/com_google_protobuf/src/google/protobuf/descriptor.cc:3915:60: warning: 'void* memset(void*, int, size_t)' clearing an object of type 'class google::protobuf::EnumValueDescriptor' with no trivial copy-assignment; use assignment or value-initialization instead [-Wclass-memaccess]
     3915 |     memset(placeholder_value, 0, sizeof(*placeholder_value));
          |                                                            ^
    In file included from external/com_google_protobuf/src/google/protobuf/message.h:121,
                     from external/com_google_protobuf/src/google/protobuf/descriptor.pb.h:30,
                     from external/com_google_protobuf/src/google/protobuf/descriptor.cc:50:
    external/com_google_protobuf/src/google/protobuf/descriptor.h:1062:23: note: 'class google::protobuf::EnumValueDescriptor' declared here
     1062 | class PROTOBUF_EXPORT EnumValueDescriptor {
          |                       ^~~~~~~~~~~~~~~~~~~
    external/com_google_protobuf/src/google/protobuf/descriptor.cc:3934:64: warning: 'void* memset(void*, int, size_t)' clearing an object of type 'class google::protobuf::Descriptor' with no trivial copy-assignment; use assignment or value-initialization instead [-Wclass-memaccess]
     3934 |     memset(placeholder_message, 0, sizeof(*placeholder_message));
          |                                                                ^
    In file included from external/com_google_protobuf/src/google/protobuf/message.h:121,
                     from external/com_google_protobuf/src/google/protobuf/descriptor.pb.h:30,
                     from external/com_google_protobuf/src/google/protobuf/descriptor.cc:50:
    external/com_google_protobuf/src/google/protobuf/descriptor.h:233:23: note: 'class google::protobuf::Descriptor' declared here
      233 | class PROTOBUF_EXPORT Descriptor {
          |                       ^~~~~~~~~~
    external/com_google_protobuf/src/google/protobuf/descriptor.cc: In member function 'google::protobuf::FileDescriptor* google::protobuf::DescriptorPool::NewPlaceholderFileWithMutexHeld(const string&) const':
    external/com_google_protobuf/src/google/protobuf/descriptor.cc:3968:46: warning: 'void* memset(void*, int, size_t)' clearing an object of type 'class google::protobuf::FileDescriptor' with no trivial copy-assignment; use assignment or value-initialization instead [-Wclass-memaccess]
     3968 |   memset(placeholder, 0, sizeof(*placeholder));
          |                                              ^
    In file included from external/com_google_protobuf/src/google/protobuf/message.h:121,
                     from external/com_google_protobuf/src/google/protobuf/descriptor.pb.h:30,
                     from external/com_google_protobuf/src/google/protobuf/descriptor.cc:50:
    external/com_google_protobuf/src/google/protobuf/descriptor.h:1309:23: note: 'class google::protobuf::FileDescriptor' declared here
     1309 | class PROTOBUF_EXPORT FileDescriptor {
          |                       ^~~~~~~~~~~~~~
    [419 / 514] Compiling external/com_google_protobuf/src/google/protobuf/descriptor.pb.cc; 6s linux-sandbox ... (12 actions, 11 running)
    ERROR: /home/matt/.cache/bazel/_bazel_matt/5655070513dc5b66b5cbb80b53498d5d/external/com_github_grpc_grpc/BUILD:545:1: C++ compilation of rule '@com_github_grpc_grpc//:gpr_base' failed (Exit 1) gcc failed: error executing command
      (cd /home/matt/.cache/bazel/_bazel_matt/5655070513dc5b66b5cbb80b53498d5d/sandbox/linux-sandbox/372/execroot/com_github_ray_project_ray && \
      exec env - \
        PATH=/home/matt/anaconda3/envs/gpu/bin:/home/matt/anaconda3/condabin:/home/matt/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin \
        PWD=/proc/self/cwd \
        PYTHON_BIN_PATH=/home/matt/anaconda3/envs/gpu/bin/python \
      /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections '-std=c++0x' -MD -MF bazel-out/k8-opt/bin/external/com_github_grpc_grpc/_objs/gpr_base/log_linux.pic.d '-frandom-seed=bazel-out/k8-opt/bin/external/com_github_grpc_grpc/_objs/gpr_base/log_linux.pic.o' -fPIC -iquote external/com_github_grpc_grpc -iquote bazel-out/k8-opt/bin/external/com_github_grpc_grpc -isystem external/com_github_grpc_grpc/include -isystem bazel-out/k8-opt/bin/external/com_github_grpc_grpc/include -isystem external/com_github_grpc_grpc/src/core/ext/upb-generated -isystem bazel-out/k8-opt/bin/external/com_github_grpc_grpc/src/core/ext/upb-generated -Werror -w -DGRPC_BAZEL_BUILD -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c external/com_github_grpc_grpc/src/core/lib/gpr/log_linux.cc -o bazel-out/k8-opt/bin/external/com_github_grpc_grpc/_objs/gpr_base/log_linux.pic.o)
    Execution platform: @bazel_tools//platforms:host_platform

    Use --sandbox_debug to see verbose messages from the sandbox: gcc failed: error executing command
      (cd /home/matt/.cache/bazel/_bazel_matt/5655070513dc5b66b5cbb80b53498d5d/sandbox/linux-sandbox/372/execroot/com_github_ray_project_ray && \
      exec env - \
        PATH=/home/matt/anaconda3/envs/gpu/bin:/home/matt/anaconda3/condabin:/home/matt/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin \
        PWD=/proc/self/cwd \
        PYTHON_BIN_PATH=/home/matt/anaconda3/envs/gpu/bin/python \
      /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections '-std=c++0x' -MD -MF bazel-out/k8-opt/bin/external/com_github_grpc_grpc/_objs/gpr_base/log_linux.pic.d '-frandom-seed=bazel-out/k8-opt/bin/external/com_github_grpc_grpc/_objs/gpr_base/log_linux.pic.o' -fPIC -iquote external/com_github_grpc_grpc -iquote bazel-out/k8-opt/bin/external/com_github_grpc_grpc -isystem external/com_github_grpc_grpc/include -isystem bazel-out/k8-opt/bin/external/com_github_grpc_grpc/include -isystem external/com_github_grpc_grpc/src/core/ext/upb-generated -isystem bazel-out/k8-opt/bin/external/com_github_grpc_grpc/src/core/ext/upb-generated -Werror -w -DGRPC_BAZEL_BUILD -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c external/com_github_grpc_grpc/src/core/lib/gpr/log_linux.cc -o bazel-out/k8-opt/bin/external/com_github_grpc_grpc/_objs/gpr_base/log_linux.pic.o)
    Execution platform: @bazel_tools//platforms:host_platform

    Use --sandbox_debug to see verbose messages from the sandbox
    external/com_github_grpc_grpc/src/core/lib/gpr/log_linux.cc:43:13: error: ambiguating new declaration of 'long int gettid()'
       43 | static long gettid(void) { return syscall(__NR_gettid); }
          |             ^~~~~~
    In file included from /usr/include/unistd.h:1170,
                     from external/com_github_grpc_grpc/src/core/lib/gpr/log_linux.cc:41:
    /usr/include/x86_64-linux-gnu/bits/unistd_ext.h:34:16: note: old declaration '__pid_t gettid()'
       34 | extern __pid_t gettid (void) __THROW;
          |                ^~~~~~
    Target //:ray_pkg failed to build
    INFO: Elapsed time: 168.147s, Critical Path: 16.70s
    INFO: 361 processes: 361 linux-sandbox.
    FAILED: Build did NOT complete successfully
    FAILED: Build did NOT complete successfully
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/home/matt/Code/ray/python/setup.py", line 198, in <module>
        license="Apache 2.0")
      File "/home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages/setuptools/__init__.py", line 145, in setup
        return distutils.core.setup(**attrs)
      File "/home/matt/anaconda3/envs/gpu/lib/python3.7/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/home/matt/anaconda3/envs/gpu/lib/python3.7/distutils/dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "/home/matt/anaconda3/envs/gpu/lib/python3.7/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages/setuptools/command/develop.py", line 38, in run
        self.install_for_development()
      File "/home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages/setuptools/command/develop.py", line 140, in install_for_development
        self.run_command('build_ext')
      File "/home/matt/anaconda3/envs/gpu/lib/python3.7/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/home/matt/anaconda3/envs/gpu/lib/python3.7/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/home/matt/Code/ray/python/setup.py", line 96, in run
        subprocess.check_call(command)
      File "/home/matt/anaconda3/envs/gpu/lib/python3.7/subprocess.py", line 347, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['../build.sh', '-p', '/home/matt/anaconda3/envs/gpu/bin/python']' returned non-zero exit status 1.
Cleaning up...
Removed build tracker '/tmp/pip-req-tracker-03xdp_qj'
ERROR: Command errored out with exit status 1: /home/matt/anaconda3/envs/gpu/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/matt/Code/ray/python/setup.py'"'"'; __file__='"'"'/home/matt/Code/ray/python/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.
Exception information:
Traceback (most recent call last):
  File "/home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 153, in _main
    status = self.run(options, args)
  File "/home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 455, in run
    use_user_site=options.use_user_site,
  File "/home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages/pip/_internal/req/__init__.py", line 62, in install_given_reqs
    **kwargs
  File "/home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 851, in install
    install_options, global_options, prefix=prefix,
  File "/home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 700, in install_editable
    cwd=self.unpacked_source_directory,
  File "/home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages/pip/_internal/utils/subprocess.py", line 242, in call_subprocess
    raise InstallationError(exc_msg)
pip._internal.exceptions.InstallationError: Command errored out with exit status 1: /home/matt/anaconda3/envs/gpu/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/matt/Code/ray/python/setup.py'"'"'; __file__='"'"'/home/matt/Code/ray/python/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.
@pcmoritz
Copy link
Contributor

pcmoritz commented Oct 23, 2019 •
edited
Loading

@mawright Thanks for the report! It is failing to compile grpc, can you try to replace this line here:
https://github.com/ray-project/ray/blob/master/bazel/ray_deps_setup.bzl#L108

and the line two below to the latest release commit https://github.com/grpc/grpc/archive/2de2e8dd8921e1f7d043e01faf7fe8a291fbb072.tar.gz and see if it is fixed in the latest grpc release?

@mawright
Copy link
Contributor Author

Hi Phillip, I was digging around and it looks like you were right about it being a bug in grpc. It looks like that bug was fixed in this commit: grpc/grpc@57586a1

Unfortunately when I updated the link to the grpc artifact like you said,

    http_archive(
        name = "com_github_grpc_grpc",
        urls = [
            "https://github.com/grpc/grpc/archive/2de2e8dd8921e1f7d043e01faf7fe8a291fbb072.tar.gz"
        ],
        strip_prefix = "grpc-2de2e8dd8921e1f7d043e01faf7fe8a291fbb072",
    )

I got a new error:

(gpu) matt@Rocko2:~/Code/ray/python$ pip install -e . --verbose
Created temporary directory: /tmp/pip-ephem-wheel-cache-7_1z5_sq
Created temporary directory: /tmp/pip-req-tracker-5lajio2g
Created requirements tracker '/tmp/pip-req-tracker-5lajio2g'
Created temporary directory: /tmp/pip-install-q6klhqg_
Obtaining file:///home/matt/Code/ray/python
  Added file:///home/matt/Code/ray/python to build tracker '/tmp/pip-req-tracker-5lajio2g'
    Running setup.py (path:/home/matt/Code/ray/python/setup.py) egg_info for package from file:///home/matt/Code/ray/python
    Running command python setup.py egg_info
    running egg_info
    writing ray.egg-info/PKG-INFO
    writing dependency_links to ray.egg-info/dependency_links.txt
    writing entry points to ray.egg-info/entry_points.txt
    writing requirements to ray.egg-info/requires.txt
    writing top-level names to ray.egg-info/top_level.txt
    reading manifest file 'ray.egg-info/SOURCES.txt'
    writing manifest file 'ray.egg-info/SOURCES.txt'
  Source in /home/matt/Code/ray/python has version 0.8.0.dev6, which satisfies requirement ray==0.8.0.dev6 from file:///home/matt/Code/ray/python
  Removed ray==0.8.0.dev6 from file:///home/matt/Code/ray/python from build tracker '/tmp/pip-req-tracker-5lajio2g'
Requirement already satisfied: numpy>=1.14 in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from ray==0.8.0.dev6) (1.17.2)
Requirement already satisfied: filelock in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from ray==0.8.0.dev6) (3.0.12)
Requirement already satisfied: jsonschema in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from ray==0.8.0.dev6) (3.1.1)
Requirement already satisfied: funcsigs in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from ray==0.8.0.dev6) (1.0.2)
Requirement already satisfied: click in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from ray==0.8.0.dev6) (7.0)
Requirement already satisfied: colorama in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from ray==0.8.0.dev6) (0.4.1)
Requirement already satisfied: pytest in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from ray==0.8.0.dev6) (5.2.1)
Requirement already satisfied: pyyaml in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from ray==0.8.0.dev6) (5.1.2)
Requirement already satisfied: redis>=3.3.2 in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from ray==0.8.0.dev6) (3.3.8)
Requirement already satisfied: six>=1.0.0 in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from ray==0.8.0.dev6) (1.12.0)
Requirement already satisfied: protobuf>=3.8.0 in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from ray==0.8.0.dev6) (3.9.2)
Requirement already satisfied: pyrsistent>=0.14.0 in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from jsonschema->ray==0.8.0.dev6) (0.15.4)
Requirement already satisfied: attrs>=17.4.0 in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from jsonschema->ray==0.8.0.dev6) (19.3.0)
Requirement already satisfied: importlib-metadata in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from jsonschema->ray==0.8.0.dev6) (0.23)
Requirement already satisfied: setuptools in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from jsonschema->ray==0.8.0.dev6) (41.4.0)
Requirement already satisfied: py>=1.5.0 in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from pytest->ray==0.8.0.dev6) (1.8.0)
Requirement already satisfied: packaging in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from pytest->ray==0.8.0.dev6) (19.2)
Requirement already satisfied: more-itertools>=4.0.0 in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from pytest->ray==0.8.0.dev6) (7.2.0)
Requirement already satisfied: atomicwrites>=1.0 in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from pytest->ray==0.8.0.dev6) (1.3.0)
Requirement already satisfied: pluggy<1.0,>=0.12 in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from pytest->ray==0.8.0.dev6) (0.13.0)
Requirement already satisfied: wcwidth in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from pytest->ray==0.8.0.dev6) (0.1.7)
Requirement already satisfied: zipp>=0.5 in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from importlib-metadata->jsonschema->ray==0.8.0.dev6) (0.6.0)
Requirement already satisfied: pyparsing>=2.0.2 in /home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages (from packaging->pytest->ray==0.8.0.dev6) (2.4.2)
Installing collected packages: ray
  Running setup.py develop for ray
    Running command /home/matt/anaconda3/envs/gpu/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/matt/Code/ray/python/setup.py'"'"'; __file__='"'"'/home/matt/Code/ray/python/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps
    running develop
    running egg_info
    writing ray.egg-info/PKG-INFO
    writing dependency_links to ray.egg-info/dependency_links.txt
    writing entry points to ray.egg-info/entry_points.txt
    writing requirements to ray.egg-info/requires.txt
    writing top-level names to ray.egg-info/top_level.txt
    reading manifest file 'ray.egg-info/SOURCES.txt'
    writing manifest file 'ray.egg-info/SOURCES.txt'
    running build_ext
    + set -e
    +++ dirname ../build.sh
    ++ cd ..
    ++ pwd
    + ROOT_DIR=/home/matt/Code/ray
    ++ uname
    + unamestr=Linux
    + [[ Linux == \L\i\n\u\x ]]
    + PARALLEL=1
    + RAY_BUILD_PYTHON=YES
    + RAY_BUILD_JAVA=NO
    + PYTHON_EXECUTABLE=
    + BUILD_DIR=
    + [[ 2 -gt 0 ]]
    + key=-p
    + case $key in
    + PYTHON_EXECUTABLE=/home/matt/anaconda3/envs/gpu/bin/python
    + shift
    + shift
    + [[ 0 -gt 0 ]]
    + [[ -z /home/matt/anaconda3/envs/gpu/bin/python ]]
    + echo 'Using Python executable /home/matt/anaconda3/envs/gpu/bin/python.'
    Using Python executable /home/matt/anaconda3/envs/gpu/bin/python.
    ++ PATH=/home/matt/anaconda3/envs/gpu/bin:/home/matt/anaconda3/condabin:/home/matt/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/matt/.bazel/bin
    ++ which bazel
    + BAZEL_EXECUTABLE=/home/matt/bin/bazel
    + echo 'Using Bazel executable /home/matt/bin/bazel.'
    Using Bazel executable /home/matt/bin/bazel.
    + RAY_BUILD_PYTHON=YES
    + RAY_BUILD_JAVA=NO
    + bash /home/matt/Code/ray/setup_thirdparty.sh /home/matt/anaconda3/envs/gpu/bin/python
    + set -e
    +++ dirname /home/matt/Code/ray/setup_thirdparty.sh
    ++ cd /home/matt/Code/ray
    ++ pwd
    + ROOT_DIR=/home/matt/Code/ray
    + [[ -z /home/matt/anaconda3/envs/gpu/bin/python ]]
    + PYTHON_EXECUTABLE=/home/matt/anaconda3/envs/gpu/bin/python
    + echo 'Using Python executable /home/matt/anaconda3/envs/gpu/bin/python.'
    Using Python executable /home/matt/anaconda3/envs/gpu/bin/python.
    + RAY_BUILD_PYTHON=YES
    + RAY_BUILD_JAVA=NO
    + /home/matt/Code/ray/thirdparty/scripts/setup.sh /home/matt/anaconda3/envs/gpu/bin/python
    + set -e
    +++ dirname /home/matt/Code/ray/thirdparty/scripts/setup.sh
    ++ cd /home/matt/Code/ray/thirdparty/scripts
    ++ pwd
    + TP_SCRIPT_DIR=/home/matt/Code/ray/thirdparty/scripts
    + TP_DIR=/home/matt/Code/ray/thirdparty/scripts/..
    + mkdir -p /home/matt/Code/ray/thirdparty/scripts/../build
    + mkdir -p /home/matt/Code/ray/thirdparty/scripts/../pkg
    + [[ -z /home/matt/anaconda3/envs/gpu/bin/python ]]
    + PYTHON_EXECUTABLE=/home/matt/anaconda3/envs/gpu/bin/python
    + echo 'Using Python executable /home/matt/anaconda3/envs/gpu/bin/python.'
    Using Python executable /home/matt/anaconda3/envs/gpu/bin/python.
    + [[ NO == \Y\E\S ]]
    + [[ YES == \Y\E\S ]]
    + echo 'Python library will be built.'
    Python library will be built.
    ++ uname
    + unamestr=Linux
    + BUILD_DIR=/home/matt/Code/ray/build/
    + '[' '!' -d /home/matt/Code/ray/build/ ']'
    + pushd /home/matt/Code/ray/build/
    ~/Code/ray/build ~/Code/ray/python
    + '[' -z '' ']'
    + /home/matt/anaconda3/envs/gpu/bin/python -m pip install -q --target=/home/matt/Code/ray/python/ray/pyarrow_files pyarrow==0.14.0.RAY --find-links https://s3-us-west-2.amazonaws.com/arrow-wheels/516e15028091b5e287200b5df77d77f72d9a6c9a/index.html
    WARNING: Target directory /home/matt/Code/ray/python/ray/pyarrow_files/pyarrow already exists. Specify --upgrade to force replacement.
    WARNING: Target directory /home/matt/Code/ray/python/ray/pyarrow_files/numpy-1.17.3.dist-info already exists. Specify --upgrade to force replacement.
    WARNING: Target directory /home/matt/Code/ray/python/ray/pyarrow_files/six-1.12.0.dist-info already exists. Specify --upgrade to force replacement.
    WARNING: Target directory /home/matt/Code/ray/python/ray/pyarrow_files/six.py already exists. Specify --upgrade to force replacement.
    WARNING: Target directory /home/matt/Code/ray/python/ray/pyarrow_files/pyarrow-0.14.0.RAY.dist-info already exists. Specify --upgrade to force replacement.
    WARNING: Target directory /home/matt/Code/ray/python/ray/pyarrow_files/__pycache__ already exists. Specify --upgrade to force replacement.
    WARNING: Target directory /home/matt/Code/ray/python/ray/pyarrow_files/numpy already exists. Specify --upgrade to force replacement.
    WARNING: Target directory /home/matt/Code/ray/python/ray/pyarrow_files/bin already exists. Specify --upgrade to force replacement.
    + export PYTHON_BIN_PATH=/home/matt/anaconda3/envs/gpu/bin/python
    + PYTHON_BIN_PATH=/home/matt/anaconda3/envs/gpu/bin/python
    + '[' NO == YES ']'
    + '[' YES == YES ']'
    + /home/matt/bin/bazel build //:ray_pkg --verbose_failures
    Extracting Bazel installation...
    Starting local Bazel server and connecting to it...
    Loading:
    Loading: 0 packages loaded
    Loading: 0 packages loaded
    DEBUG: Rule 'com_github_checkstyle_java' indicated that a canonical reproducible form can be obtained by modifying arguments shallow_since = "1552542575 +0800"
    DEBUG: Rule 'com_github_nelhage_rules_boost' indicated that a canonical reproducible form can be obtained by modifying arguments shallow_since = "1556014830 +0800"
    INFO: SHA256 (https://github.com/jovany-wang/prometheus-cpp/archive/master.zip) = d0c773da8af3db99c543dd0413f4427d835170eddfd517bfeba104236a8d2c07
    DEBUG: Rule 'com_github_jupp0r_prometheus_cpp' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "d0c773da8af3db99c543dd0413f4427d835170eddfd517bfeba104236a8d2c07"
    Loading: 0 packages loaded
    Loading: 0 packages loaded
    Loading: 0 packages loaded
    Loading: 0 packages loaded
    Loading: 0 packages loaded
    Loading: 0 packages loaded
    Loading: 0 packages loaded
    INFO: SHA256 (https://github.com/grpc/grpc/archive/2de2e8dd8921e1f7d043e01faf7fe8a291fbb072.tar.gz) = c396cabdf18650a61c0ecb171fec8bae6580f686b52558f59f410809bdbe0bae
    Loading: 0 packages loaded
    DEBUG: Rule 'com_github_grpc_grpc' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "c396cabdf18650a61c0ecb171fec8bae6580f686b52558f59f410809bdbe0bae"
    INFO: SHA256 (https://github.com/google/bazel-common/archive/f1115e0f777f08c3cdb115526c4e663005bec69b.zip) = 1e05a4791cc3470d3ecf7edb556f796b1d340359f1c4d293f175d4d0946cf84c
    DEBUG: Rule 'bazel_common' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "1e05a4791cc3470d3ecf7edb556f796b1d340359f1c4d293f175d4d0946cf84c"
    INFO: SHA256 (https://github.com/bazelbuild/bazel-skylib/archive/0.6.0.tar.gz) = eb5c57e4c12e68c0c20bc774bfbc60a568e800d025557bc4ea022c6479acc867
    DEBUG: Rule 'bazel_skylib' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "eb5c57e4c12e68c0c20bc774bfbc60a568e800d025557bc4ea022c6479acc867"
    Loading: 0 packages loaded
        currently loading:
    Loading: 0 packages loaded
        currently loading:
    DEBUG: Rule 'com_github_google_flatbuffers' indicated that a canonical reproducible form can be obtained by modifying arguments shallow_since = "1547755012 -0800"
    Analyzing: target //:ray_pkg (1 packages loaded, 0 targets configured)
    Analyzing: target //:ray_pkg (3 packages loaded, 16 targets configured)
    Analyzing: target //:ray_pkg (3 packages loaded, 16 targets configured)
    Analyzing: target //:ray_pkg (3 packages loaded, 16 targets configured)
    DEBUG: Rule 'plasma' indicated that a canonical reproducible form can be obtained by modifying arguments shallow_since = "1570611764 -0700"
    INFO: Call stack for the definition of repository 'boost' which is a http_archive (rule definition at /home/matt/.cache/bazel/_bazel_matt/5655070513dc5b66b5cbb80b53498d5d/external/bazel_tools/tools/build_defs/repo/http.bzl:229:16):
     - /home/matt/.cache/bazel/_bazel_matt/5655070513dc5b66b5cbb80b53498d5d/external/com_github_nelhage_rules_boost/boost/boost.bzl:125:9
     - /home/matt/Code/ray/bazel/ray_deps_build_all.bzl:14:3
     - /home/matt/Code/ray/WORKSPACE:9:1
    INFO: Call stack for the definition of repository 'io_opencensus_cpp' which is a http_archive (rule definition at /home/matt/.cache/bazel/_bazel_matt/5655070513dc5b66b5cbb80b53498d5d/external/bazel_tools/tools/build_defs/repo/http.bzl:229:16):
     - /home/matt/Code/ray/bazel/ray_deps_setup.bzl:81:5
     - /home/matt/Code/ray/WORKSPACE:5:1
    INFO: Call stack for the definition of repository 'com_google_protobuf' which is a http_archive (rule definition at /home/matt/.cache/bazel/_bazel_matt/5655070513dc5b66b5cbb80b53498d5d/external/bazel_tools/tools/build_defs/repo/http.bzl:229:16):
     - /home/matt/.cache/bazel/_bazel_matt/5655070513dc5b66b5cbb80b53498d5d/external/com_github_grpc_grpc/bazel/grpc_deps.bzl:120:9
     - /home/matt/Code/ray/bazel/ray_deps_build_all.bzl:17:3
     - /home/matt/Code/ray/WORKSPACE:9:1
    INFO: Call stack for the definition of repository 'rules_python' which is a http_archive (rule definition at /home/matt/.cache/bazel/_bazel_matt/5655070513dc5b66b5cbb80b53498d5d/external/bazel_tools/tools/build_defs/repo/http.bzl:229:16):
     - /home/matt/.cache/bazel/_bazel_matt/5655070513dc5b66b5cbb80b53498d5d/external/com_github_grpc_grpc/bazel/grpc_python_deps.bzl:52:9
     - /home/matt/.cache/bazel/_bazel_matt/5655070513dc5b66b5cbb80b53498d5d/external/com_github_grpc_grpc/bazel/grpc_deps.bzl:231:5
     - /home/matt/Code/ray/bazel/ray_deps_build_all.bzl:17:3
     - /home/matt/Code/ray/WORKSPACE:9:1
    INFO: Call stack for the definition of repository 'build_bazel_rules_apple' which is a http_archive (rule definition at /home/matt/.cache/bazel/_bazel_matt/5655070513dc5b66b5cbb80b53498d5d/external/bazel_tools/tools/build_defs/repo/http.bzl:229:16):
     - /home/matt/.cache/bazel/_bazel_matt/5655070513dc5b66b5cbb80b53498d5d/external/com_github_grpc_grpc/bazel/grpc_deps.bzl:224:9
     - /home/matt/Code/ray/bazel/ray_deps_build_all.bzl:17:3
     - /home/matt/Code/ray/WORKSPACE:9:1
    ERROR: /home/matt/Code/ray/BUILD.bazel:216:1: error loading package '@com_github_grpc_grpc//': in /home/matt/.cache/bazel/_bazel_matt/5655070513dc5b66b5cbb80b53498d5d/external/com_github_grpc_grpc/bazel/grpc_build_system.bzl: in /home/matt/.cache/bazel/_bazel_matt/5655070513dc5b66b5cbb80b53498d5d/external/upb/bazel/upb_proto_library.bzl: Unable to find package for @bazel_version//:bazel_version.bzl: The repository '@bazel_version' could not be resolved. and referenced by '//:ray_common'
    ERROR: Analysis of target '//:ray_pkg' failed; build aborted: error loading package '@com_github_grpc_grpc//': in /home/matt/.cache/bazel/_bazel_matt/5655070513dc5b66b5cbb80b53498d5d/external/com_github_grpc_grpc/bazel/grpc_build_system.bzl: in /home/matt/.cache/bazel/_bazel_matt/5655070513dc5b66b5cbb80b53498d5d/external/upb/bazel/upb_proto_library.bzl: Unable to find package for @bazel_version//:bazel_version.bzl: The repository '@bazel_version' could not be resolved.
    INFO: Elapsed time: 75.974s
    INFO: 0 processes.
    FAILED: Build did NOT complete successfully (14 packages loaded, 168 targets configured)
    FAILED: Build did NOT complete successfully (14 packages loaded, 168 targets configured)
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/home/matt/Code/ray/python/setup.py", line 198, in <module>
        license="Apache 2.0")
      File "/home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages/setuptools/__init__.py", line 145, in setup
        return distutils.core.setup(**attrs)
      File "/home/matt/anaconda3/envs/gpu/lib/python3.7/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/home/matt/anaconda3/envs/gpu/lib/python3.7/distutils/dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "/home/matt/anaconda3/envs/gpu/lib/python3.7/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages/setuptools/command/develop.py", line 38, in run
        self.install_for_development()
      File "/home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages/setuptools/command/develop.py", line 140, in install_for_development
        self.run_command('build_ext')
      File "/home/matt/anaconda3/envs/gpu/lib/python3.7/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/home/matt/anaconda3/envs/gpu/lib/python3.7/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/home/matt/Code/ray/python/setup.py", line 96, in run
        subprocess.check_call(command)
      File "/home/matt/anaconda3/envs/gpu/lib/python3.7/subprocess.py", line 347, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['../build.sh', '-p', '/home/matt/anaconda3/envs/gpu/bin/python']' returned non-zero exit status 1.
Cleaning up...
Removed build tracker '/tmp/pip-req-tracker-5lajio2g'
ERROR: Command errored out with exit status 1: /home/matt/anaconda3/envs/gpu/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/matt/Code/ray/python/setup.py'"'"'; __file__='"'"'/home/matt/Code/ray/python/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.
Exception information:
Traceback (most recent call last):
  File "/home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 153, in _main
    status = self.run(options, args)
  File "/home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 455, in run
    use_user_site=options.use_user_site,
  File "/home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages/pip/_internal/req/__init__.py", line 62, in install_given_reqs
    **kwargs
  File "/home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 851, in install
    install_options, global_options, prefix=prefix,
  File "/home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 700, in install_editable
    cwd=self.unpacked_source_directory,
  File "/home/matt/anaconda3/envs/gpu/lib/python3.7/site-packages/pip/_internal/utils/subprocess.py", line 242, in call_subprocess
    raise InstallationError(exc_msg)
pip._internal.exceptions.InstallationError: Command errored out with exit status 1: /home/matt/anaconda3/envs/gpu/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/matt/Code/ray/python/setup.py'"'"'; __file__='"'"'/home/matt/Code/ray/python/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.

@pcmoritz
Copy link
Contributor

Thanks for trying it out! This seems to be a new issue that is fixed in grpc/grpc#20586, however with that PR I get another error:

ERROR: /Users/pcmoritz/ray/BUILD.bazel:216:1: error loading package '@com_github_grpc_grpc//': in /private/var/tmp/_bazel_pcmoritz/65b142b3f2651cc0f42773a358f37553/external/com_github_grpc_grpc/bazel/grpc_build_system.bzl: in /private/var/tmp/_bazel_pcmoritz/65b142b3f2651cc0f42773a358f37553/external/build_bazel_rules_apple/apple/ios.bzl: in /private/var/tmp/_bazel_pcmoritz/65b142b3f2651cc0f42773a358f37553/external/build_bazel_rules_apple/apple/internal/ios_rules.bzl: in /private/var/tmp/_bazel_pcmoritz/65b142b3f2651cc0f42773a358f37553/external/build_bazel_rules_apple/apple/internal/rule_factory.bzl: Unable to find package for @build_bazel_rules_swift//swift:swift.bzl: The repository '@build_bazel_rules_swift' could not be resolved. and referenced by '//:ray_common'
ERROR: Analysis of target '//:ray_pkg' failed; build aborted: error loading package '@com_github_grpc_grpc//': in /private/var/tmp/_bazel_pcmoritz/65b142b3f2651cc0f42773a358f37553/external/com_github_grpc_grpc/bazel/grpc_build_system.bzl: in /private/var/tmp/_bazel_pcmoritz/65b142b3f2651cc0f42773a358f37553/external/build_bazel_rules_apple/apple/ios.bzl: in /private/var/tmp/_bazel_pcmoritz/65b142b3f2651cc0f42773a358f37553/external/build_bazel_rules_apple/apple/internal/ios_rules.bzl: in /private/var/tmp/_bazel_pcmoritz/65b142b3f2651cc0f42773a358f37553/external/build_bazel_rules_apple/apple/internal/rule_factory.bzl: Unable to find package for @build_bazel_rules_swift//swift:swift.bzl: The repository '@build_bazel_rules_swift' could not be resolved.

Is it possible for you to use ubuntu 18 in the meantime while this gets fixed?

@pcmoritz pcmoritz changed the title Ray doesn't build Ray doesn't build on 19.10 due to GRPC Oct 24, 2019
@pcmoritz pcmoritz changed the title Ray doesn't build on 19.10 due to GRPC Ray doesn't build on Ubuntu 19.10 due to GRPC Oct 24, 2019
@gallmerci
Copy link

Did you call grpc_extra_deps() introduced in grpc/grpc#20586 from somewhere here?
It seems that apple_rules_dependencies() is not called which sets up the build_bazel_rules_swift repo.

@mawright
Copy link
Contributor Author

Thanks for trying it out! This seems to be a new issue that is fixed in grpc/grpc#20586, however with that PR I get another error:

ERROR: /Users/pcmoritz/ray/BUILD.bazel:216:1: error loading package '@com_github_grpc_grpc//': in /private/var/tmp/_bazel_pcmoritz/65b142b3f2651cc0f42773a358f37553/external/com_github_grpc_grpc/bazel/grpc_build_system.bzl: in /private/var/tmp/_bazel_pcmoritz/65b142b3f2651cc0f42773a358f37553/external/build_bazel_rules_apple/apple/ios.bzl: in /private/var/tmp/_bazel_pcmoritz/65b142b3f2651cc0f42773a358f37553/external/build_bazel_rules_apple/apple/internal/ios_rules.bzl: in /private/var/tmp/_bazel_pcmoritz/65b142b3f2651cc0f42773a358f37553/external/build_bazel_rules_apple/apple/internal/rule_factory.bzl: Unable to find package for @build_bazel_rules_swift//swift:swift.bzl: The repository '@build_bazel_rules_swift' could not be resolved. and referenced by '//:ray_common'
ERROR: Analysis of target '//:ray_pkg' failed; build aborted: error loading package '@com_github_grpc_grpc//': in /private/var/tmp/_bazel_pcmoritz/65b142b3f2651cc0f42773a358f37553/external/com_github_grpc_grpc/bazel/grpc_build_system.bzl: in /private/var/tmp/_bazel_pcmoritz/65b142b3f2651cc0f42773a358f37553/external/build_bazel_rules_apple/apple/ios.bzl: in /private/var/tmp/_bazel_pcmoritz/65b142b3f2651cc0f42773a358f37553/external/build_bazel_rules_apple/apple/internal/ios_rules.bzl: in /private/var/tmp/_bazel_pcmoritz/65b142b3f2651cc0f42773a358f37553/external/build_bazel_rules_apple/apple/internal/rule_factory.bzl: Unable to find package for @build_bazel_rules_swift//swift:swift.bzl: The repository '@build_bazel_rules_swift' could not be resolved.

Is it possible for you to use ubuntu 18 in the meantime while this gets fixed?

I am only doing development on Python stuff so I think I will be able to work around it by installing from the wheel and symlinking stuff.

@pcmoritz pcmoritz mentioned this issue Oct 24, 2019
Merged
3 tasks
@pcmoritz
Copy link
Contributor

@gallmerci Thanks for your help! I tried what you suggested (in #6002) and now I get the following error:

ERROR: Failed to load Starlark extension '@envoy_api//bazel:repositories.bzl'.
Cycle in the workspace file detected. This indicates that a repository is used prior to being defined.
The following chain of repository dependencies lead to the missing definition.
 - @envoy_api
This could either mean you have to add the '@envoy_api' repository with a statement like `http_archive` in your WORKSPACE file (note that transitive dependencies are not added automatically), or move an existing definition earlier in your WORKSPACE file.

So it looks like grpc_extra_deps still doesn't contain everything needed.

@gallmerci
Copy link

@pcmoritz you need to execute grpc_deps() before you load grpc_extra_deps.bzl because the repos are defined after calling grpc_deps()

load("@com_github_grpc_grpc//bazel:grpc_deps.bzl", "grpc_deps", "grpc_test_only_deps")
grpc_deps()
load("@com_github_grpc_grpc//bazel:grpc_extra_deps.bzl", "grpc_extra_deps")
grpc_extra_deps()

@pcmoritz
Copy link
Contributor

@gallmerci That indeed fixes the problem, thank you so much for your help. I updated the PR.

@pcmoritz pcmoritz mentioned this issue Oct 25, 2019
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Use GRCP and Bazel 1.0 pcmoritz/ray-1
3 participants
@pcmoritz @mawright @gallmerci