Segfault when importing TensorFlow after Ray.

[robertnishihara]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04.4 LTS
• Ray installed from (source or binary): pip from ray-wheels bucket (somewhere close to 6172f94)
• Ray version: 0.4.0
• Python version: Python 3.6.1 :: Continuum Analytics, Inc.
• Exact command to reproduce:

import ray
import tensorflow  # This line segfaults.

TensorFlow version 1.8.0.

Searching over commits, the issue was introduced in 225608e.

gdb shows

(gdb) run
Starting program: /opt/conda/bin/python 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Python 3.6.1 |Continuum Analytics, Inc.| (default, May 11 2017, 13:09:58) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ray
import t[New Thread 0x7fffeda4a700 (LWP 7075)]
[New Thread 0x7fffed249700 (LWP 7076)]
[New Thread 0x7fffe8a48700 (LWP 7077)]
[New Thread 0x7fffe6247700 (LWP 7078)]
[New Thread 0x7fffe3a46700 (LWP 7079)]
[New Thread 0x7fffe1245700 (LWP 7080)]
[New Thread 0x7fffdea44700 (LWP 7081)]
[New Thread 0x7fffdc243700 (LWP 7082)]
[New Thread 0x7fffd9a42700 (LWP 7083)]
[New Thread 0x7fffd7241700 (LWP 7084)]
[New Thread 0x7fffd4a40700 (LWP 7085)]
[New Thread 0x7fffd223f700 (LWP 7086)]
[New Thread 0x7fffcfa3e700 (LWP 7087)]
[New Thread 0x7fffcd23d700 (LWP 7088)]
[New Thread 0x7fffcaa3c700 (LWP 7089)]
[New Thread 0x7fffca23b700 (LWP 7090)]
[New Thread 0x7fffc5a3a700 (LWP 7091)]
[New Thread 0x7fffc3239700 (LWP 7092)]
[New Thread 0x7fffc0a38700 (LWP 7093)]
[New Thread 0x7fffbe237700 (LWP 7094)]
[New Thread 0x7fffbba36700 (LWP 7095)]
[New Thread 0x7fffb9235700 (LWP 7096)]
[New Thread 0x7fffb6a34700 (LWP 7097)]
[Thread 0x7fffe6247700 (LWP 7078) exited]
[Thread 0x7fffd4a40700 (LWP 7085) exited]
[Thread 0x7fffcd23d700 (LWP 7088) exited]
[Thread 0x7fffbba36700 (LWP 7095) exited]
[Thread 0x7fffbe237700 (LWP 7094) exited]
[Thread 0x7fffc0a38700 (LWP 7093) exited]
[Thread 0x7fffc3239700 (LWP 7092) exited]
[Thread 0x7fffc5a3a700 (LWP 7091) exited]
[Thread 0x7fffb9235700 (LWP 7096) exited]
[Thread 0x7fffb6a34700 (LWP 7097) exited]
[Thread 0x7fffca23b700 (LWP 7090) exited]
[Thread 0x7fffcaa3c700 (LWP 7089) exited]
[Thread 0x7fffcfa3e700 (LWP 7087) exited]
[Thread 0x7fffd223f700 (LWP 7086) exited]
[Thread 0x7fffd7241700 (LWP 7084) exited]
[Thread 0x7fffd9a42700 (LWP 7083) exited]
[Thread 0x7fffdc243700 (LWP 7082) exited]
[Thread 0x7fffdea44700 (LWP 7081) exited]
[Thread 0x7fffe1245700 (LWP 7080) exited]
[Thread 0x7fffe3a46700 (LWP 7079) exited]
[Thread 0x7fffe8a48700 (LWP 7077) exited]
[Thread 0x7fffed249700 (LWP 7076) exited]
[Thread 0x7fffeda4a700 (LWP 7075) exited]
e>>> import tensorflow

Thread 1 "python" received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x00007ffff76c1a99 in __pthread_once_slow (
    once_control=0x7fffcaa33650 <tensorflow::port::(anonymous namespace)::cpuid_once_flag>, init_routine=0x7ffff10fbfe1 <std::__once_proxy()>)
    at pthread_once.c:116
#2  0x00007fffca032f8a in void std::call_once<void (&)()>(std::once_flag&, void (&)()) ()
   from /opt/conda/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so
#3  0x00007fffca032fce in tensorflow::port::TestCPUFeature(tensorflow::port::CPUFeature) ()
   from /opt/conda/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so
#4  0x00007fffc9e44f71 in _GLOBAL__sub_I_cpu_feature_guard.cc ()
   from /opt/conda/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so
#5  0x00007ffff7de76ba in call_init (l=<optimized out>, argc=argc@entry=1, 
    argv=argv@entry=0x7fffffffe1e8, env=env@entry=0x7e0240) at dl-init.c:72
#6  0x00007ffff7de77cb in call_init (env=0x7e0240, argv=0x7fffffffe1e8, 
    argc=1, l=<optimized out>) at dl-init.c:30
---Type <return> to continue, or q <return> to quit---
#7  _dl_init (main_map=main_map@entry=0xfee720, argc=1, argv=0x7fffffffe1e8, 
    env=0x7e0240) at dl-init.c:120
#8  0x00007ffff7dec8e2 in dl_open_worker (a=a@entry=0x7fffffff78c0)
    at dl-open.c:575
#9  0x00007ffff7de7564 in _dl_catch_error (
    objname=objname@entry=0x7fffffff78b0, 
    errstring=errstring@entry=0x7fffffff78b8, 
    mallocedp=mallocedp@entry=0x7fffffff78af, 
    operate=operate@entry=0x7ffff7dec4d0 <dl_open_worker>, 
    args=args@entry=0x7fffffff78c0) at dl-error.c:187
#10 0x00007ffff7debda9 in _dl_open (
    file=0x7fffed33a848 "/opt/conda/lib/python3.6/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so", mode=-2147483646, 
    caller_dlopen=0x7ffff7a61c83 <_PyImport_FindSharedFuncptr+115>, nsid=-2, 
    argc=<optimized out>, argv=<optimized out>, env=0x7e0240) at dl-open.c:660
#11 0x00007ffff74aff09 in dlopen_doit (a=a@entry=0x7fffffff7af0) at dlopen.c:66
#12 0x00007ffff7de7564 in _dl_catch_error (objname=0x624820, 
    errstring=0x624828, mallocedp=0x624818, 
    operate=0x7ffff74afeb0 <dlopen_doit>, args=0x7fffffff7af0)
    at dl-error.c:187
---Type <return> to continue, or q <return> to quit---
#13 0x00007ffff74b0571 in _dlerror_run (
    operate=operate@entry=0x7ffff74afeb0 <dlopen_doit>, 
    args=args@entry=0x7fffffff7af0) at dlerror.c:163
#14 0x00007ffff74affa1 in __dlopen (file=<optimized out>, mode=<optimized out>)
    at dlopen.c:87
#15 0x00007ffff7a61c83 in _PyImport_FindSharedFuncptr (
    prefix=0x7ffff7ae861e "PyInit", 
    shortname=0x7fffed30c610 "_pywrap_tensorflow_internal", 
    pathname=0x7fffed33a848 "/opt/conda/lib/python3.6/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so", fp=0x0) at ./Python/dynload_shlib.c:95
#16 0x00007ffff7a3cac7 in _PyImport_LoadDynamicModuleWithSpec (
    spec=0x7fffed30b278, fp=0x0) at ./Python/importdl.c:124
#17 0x00007ffff7a398ff in _imp_create_dynamic_impl (file=<optimized out>, 
    spec=0x7fffed30b278, module=<optimized out>) at Python/import.c:2004
#18 _imp_create_dynamic (module=<optimized out>, args=<optimized out>)
    at Python/clinic/import.c.h:289
#19 0x00007ffff7993199 in PyCFunction_Call (func=0x7ffff6813ea0, 
    args=0x7fffed30b208, kwds=<optimized out>) at Objects/methodobject.c:114
#20 0x00007ffff7a1ecc2 in do_call_core (kwdict=0x7fffed30e240, 
    callargs=<optimized out>, func=0x7ffff6813ea0) at Python/ceval.c:5063
---Type <return> to continue, or q <return> to quit---
#21 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at Python/ceval.c:3366
#22 0x00007ffff7a16a60 in _PyEval_EvalCodeWithName (_co=0x7ffff684ae40, 
    globals=<optimized out>, locals=<optimized out>, args=<optimized out>, 
    argcount=2, kwnames=0x0, kwargs=0xc5fa20, kwcount=0, kwstep=1, defs=0x0, 
    defcount=0, kwdefs=0x0, closure=0x0, name=0x7ffff67f79e0, 
    qualname=0x7ffff67f79e0) at Python/ceval.c:4128
#23 0x00007ffff7a1848a in fast_function (kwnames=<optimized out>, nargs=2, 
    stack=<optimized out>, func=0x7ffff6835ea0) at Python/ceval.c:4939
#24 call_function (pp_stack=0x7fffffff8268, oparg=<optimized out>, 
    kwnames=<optimized out>) at Python/ceval.c:4819
#25 0x00007ffff7a1b15d in _PyEval_EvalFrameDefault (f=<optimized out>, 
    throwflag=<optimized out>) at Python/ceval.c:3284
#26 0x00007ffff7a15e74 in _PyFunction_FastCall (co=<optimized out>, 
    args=<optimized out>, nargs=2, globals=<optimized out>)
    at Python/ceval.c:4880
#27 0x00007ffff7a185e8 in fast_function (kwnames=0x0, nargs=2, 
    stack=<optimized out>, func=0x7ffff67bd6a8) at Python/ceval.c:4915
#28 call_function (pp_stack=0x7fffffff8498, oparg=<optimized out>, kwnames=0x0)
    at Python/ceval.c:4819
---Type <return> to continue, or q <return> to quit---
#29 0x00007ffff7a1b15d in _PyEval_EvalFrameDefault (f=<optimized out>, 
    throwflag=<optimized out>) at Python/ceval.c:3284
#30 0x00007ffff7a15e74 in _PyFunction_FastCall (co=<optimized out>, 
    args=<optimized out>, nargs=1, globals=<optimized out>)
    at Python/ceval.c:4880
#31 0x00007ffff7a185e8 in fast_function (kwnames=0x0, nargs=1, 
    stack=<optimized out>, func=0x7ffff68159d8) at Python/ceval.c:4915
#32 call_function (pp_stack=0x7fffffff86c8, oparg=<optimized out>, kwnames=0x0)
    at Python/ceval.c:4819
#33 0x00007ffff7a1b15d in _PyEval_EvalFrameDefault (f=<optimized out>, 
    throwflag=<optimized out>) at Python/ceval.c:3284
#34 0x00007ffff7a15e74 in _PyFunction_FastCall (co=<optimized out>, 
    args=<optimized out>, nargs=1, globals=<optimized out>)
    at Python/ceval.c:4880
#35 0x00007ffff7a185e8 in fast_function (kwnames=0x0, nargs=1, 
    stack=<optimized out>, func=0x7ffff6815bf8) at Python/ceval.c:4915
#36 call_function (pp_stack=0x7fffffff88f8, oparg=<optimized out>, kwnames=0x0)
    at Python/ceval.c:4819
#37 0x00007ffff7a1b15d in _PyEval_EvalFrameDefault (f=<optimized out>, 
    throwflag=<optimized out>) at Python/ceval.c:3284
---Type <return> to continue, or q <return> to quit---
#38 0x00007ffff7a15e74 in _PyFunction_FastCall (co=<optimized out>, 
    args=<optimized out>, nargs=1, globals=<optimized out>)
    at Python/ceval.c:4880
#39 0x00007ffff7a185e8 in fast_function (kwnames=0x0, nargs=1, 
    stack=<optimized out>, func=0x7ffff6815c80) at Python/ceval.c:4915
#40 call_function (pp_stack=0x7fffffff8b28, oparg=<optimized out>, kwnames=0x0)
    at Python/ceval.c:4819
#41 0x00007ffff7a1b15d in _PyEval_EvalFrameDefault (f=<optimized out>, 
    throwflag=<optimized out>) at Python/ceval.c:3284
#42 0x00007ffff7a16a60 in _PyEval_EvalCodeWithName (_co=0x7fffad8e4540, 
    globals=<optimized out>, locals=<optimized out>, args=<optimized out>, 
    argcount=3, kwnames=0x0, kwargs=0xfd64a8, kwcount=0, kwstep=1, 
    defs=0x7fffadb73c68, defcount=1, kwdefs=0x0, closure=0x0, 
    name=0x7fffadb6ee70, qualname=0x7fffadb6ee70) at Python/ceval.c:4128
#43 0x00007ffff7a1848a in fast_function (kwnames=<optimized out>, nargs=3, 
    stack=<optimized out>, func=0x7fffadb75ae8) at Python/ceval.c:4939
#44 call_function (pp_stack=0x7fffffff8dc8, oparg=<optimized out>, 
    kwnames=<optimized out>) at Python/ceval.c:4819
#45 0x00007ffff7a1b15d in _PyEval_EvalFrameDefault (f=<optimized out>, 
    throwflag=<optimized out>) at Python/ceval.c:3284
---Type <return> to continue, or q <return> to quit---
#46 0x00007ffff7a15e74 in _PyFunction_FastCall (co=<optimized out>, 
    args=<optimized out>, nargs=4, globals=<optimized out>)
    at Python/ceval.c:4880
#47 0x00007ffff7a185e8 in fast_function (kwnames=0x0, nargs=4, 
    stack=<optimized out>, func=0x7fffadb758c8) at Python/ceval.c:4915
#48 call_function (pp_stack=0x7fffffff8ff8, oparg=<optimized out>, kwnames=0x0)
    at Python/ceval.c:4819
#49 0x00007ffff7a1b15d in _PyEval_EvalFrameDefault (f=<optimized out>, 
    throwflag=<optimized out>) at Python/ceval.c:3284
#50 0x00007ffff7a15e74 in _PyFunction_FastCall (co=<optimized out>, 
    args=<optimized out>, nargs=0, globals=<optimized out>)
    at Python/ceval.c:4880
#51 0x00007ffff7a185e8 in fast_function (kwnames=0x0, nargs=0, 
    stack=<optimized out>, func=0x7fffed346510) at Python/ceval.c:4915
#52 call_function (pp_stack=0x7fffffff9228, oparg=<optimized out>, kwnames=0x0)
    at Python/ceval.c:4819
#53 0x00007ffff7a1b15d in _PyEval_EvalFrameDefault (f=<optimized out>, 
    throwflag=<optimized out>) at Python/ceval.c:3284
#54 0x00007ffff7a16a60 in _PyEval_EvalCodeWithName (_co=0x7fffed30d030, 
    globals=<optimized out>, locals=<optimized out>, args=<optimized out>, 
---Type <return> to continue, or q <return> to quit---
    argcount=0, kwnames=0x0, kwargs=0x8, kwcount=0, kwstep=2, defs=0x0, 
    defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0)
    at Python/ceval.c:4128
#55 0x00007ffff7a16ee3 in PyEval_EvalCodeEx (_co=<optimized out>, 
    globals=<optimized out>, locals=<optimized out>, args=<optimized out>, 
    argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, 
    defcount=0, kwdefs=0x0, closure=0x0) at Python/ceval.c:4149
#56 0x00007ffff7a16f2b in PyEval_EvalCode (co=<optimized out>, 
    globals=<optimized out>, locals=<optimized out>) at Python/ceval.c:695
#57 0x00007ffff7a1496e in builtin_exec_impl (locals=0x7fffed34f120, 
    globals=0x7fffed34f120, source=0x7fffed30d030, module=<optimized out>)
    at Python/bltinmodule.c:974
#58 builtin_exec (module=<optimized out>, args=<optimized out>)
    at Python/clinic/bltinmodule.c.h:281
#59 0x00007ffff7993199 in PyCFunction_Call (func=0x7ffff68569d8, 
    args=0x7fffed30c508, kwds=<optimized out>) at Objects/methodobject.c:114
#60 0x00007ffff7a1ecc2 in do_call_core (kwdict=0x7fffed34f288, 
    callargs=<optimized out>, func=0x7ffff68569d8) at Python/ceval.c:5063
#61 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at Python/ceval.c:3366
---Type <return> to continue, or q <return> to quit---
#62 0x00007ffff7a16a60 in _PyEval_EvalCodeWithName (_co=0x7ffff684ae40, 
    globals=<optimized out>, locals=<optimized out>, args=<optimized out>, 
    argcount=3, kwnames=0x0, kwargs=0x7ffff258d3c8, kwcount=0, kwstep=1, 
    defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x7ffff67f79e0, 
    qualname=0x7ffff67f79e0) at Python/ceval.c:4128
#63 0x00007ffff7a1848a in fast_function (kwnames=<optimized out>, nargs=3, 
    stack=<optimized out>, func=0x7ffff6835ea0) at Python/ceval.c:4939
#64 call_function (pp_stack=0x7fffffff9808, oparg=<optimized out>, 
    kwnames=<optimized out>) at Python/ceval.c:4819
#65 0x00007ffff7a1b15d in _PyEval_EvalFrameDefault (f=<optimized out>, 
    throwflag=<optimized out>) at Python/ceval.c:3284
#66 0x00007ffff7a15e74 in _PyFunction_FastCall (co=<optimized out>, 
    args=<optimized out>, nargs=2, globals=<optimized out>)
    at Python/ceval.c:4880
#67 0x00007ffff7a185e8 in fast_function (kwnames=0x0, nargs=2, 
    stack=<optimized out>, func=0x7ffff67bc8c8) at Python/ceval.c:4915
#68 call_function (pp_stack=0x7fffffff9a38, oparg=<optimized out>, kwnames=0x0)
    at Python/ceval.c:4819
#69 0x00007ffff7a1b15d in _PyEval_EvalFrameDefault (f=<optimized out>, 
    throwflag=<optimized out>) at Python/ceval.c:3284
---Type <return> to continue, or q <return> to quit---
#70 0x00007ffff7a15e74 in _PyFunction_FastCall (co=<optimized out>, 
    args=<optimized out>, nargs=1, globals=<optimized out>)
    at Python/ceval.c:4880
#71 0x00007ffff7a185e8 in fast_function (kwnames=0x0, nargs=1, 
    stack=<optimized out>, func=0x7ffff6815bf8) at Python/ceval.c:4915
#72 call_function (pp_stack=0x7fffffff9c68, oparg=<optimized out>, kwnames=0x0)
    at Python/ceval.c:4819
#73 0x00007ffff7a1b15d in _PyEval_EvalFrameDefault (f=<optimized out>, 
    throwflag=<optimized out>) at Python/ceval.c:3284
#74 0x00007ffff7a15e74 in _PyFunction_FastCall (co=<optimized out>, 
    args=<optimized out>, nargs=2, globals=<optimized out>)
    at Python/ceval.c:4880
#75 0x00007ffff7a185e8 in fast_function (kwnames=0x0, nargs=2, 
    stack=<optimized out>, func=0x7ffff6818d08) at Python/ceval.c:4915
#76 call_function (pp_stack=0x7fffffff9e98, oparg=<optimized out>, kwnames=0x0)
    at Python/ceval.c:4819
#77 0x00007ffff7a1b15d in _PyEval_EvalFrameDefault (f=<optimized out>, 
    throwflag=<optimized out>) at Python/ceval.c:3284
#78 0x00007ffff7a15e74 in _PyFunction_FastCall (co=<optimized out>, 
    args=<optimized out>, nargs=2, globals=<optimized out>)
---Type <return> to continue, or q <return> to quit---
    at Python/ceval.c:4880
#79 0x00007ffff7a16e75 in _PyFunction_FastCallDict (func=0x7ffff6818d90, 
    args=0x7fffffffa060, nargs=2, kwargs=0x0) at Python/ceval.c:4982
#80 0x00007ffff793bba6 in _PyObject_FastCallDict (func=0x7ffff6818d90, 
    args=0x7fffffffa060, nargs=<optimized out>, kwargs=0x0)
    at Objects/abstract.c:2295
#81 0x00007ffff793ef4f in _PyObject_CallMethodIdObjArgs (
    callable=0x7ffff6818d90, name=<optimized out>) at Objects/abstract.c:2780
#82 0x00007ffff7a3ac11 in PyImport_ImportModuleLevelObject (
    name=0x7fffed344210, globals=<optimized out>, locals=<optimized out>, 
    fromlist=0x7fffed34c0f0, level=0) at Python/import.c:1592
#83 0x00007ffff7a2046d in import_name (level=0x7ffff7da79a0 <small_ints+160>, 
    fromlist=0x7fffed34c0f0, name=0x7fffed344210, f=0xf77ac8)
    at Python/ceval.c:5176
#84 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at Python/ceval.c:2848
#85 0x00007ffff7a16a60 in _PyEval_EvalCodeWithName (_co=0x7ffff6677030, 
    globals=<optimized out>, locals=<optimized out>, args=<optimized out>, 
    argcount=0, kwnames=0x0, kwargs=0x8, kwcount=0, kwstep=2, defs=0x0, 
    defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0)
---Type <return> to continue, or q <return> to quit---
    at Python/ceval.c:4128
#86 0x00007ffff7a16ee3 in PyEval_EvalCodeEx (_co=<optimized out>, 
    globals=<optimized out>, locals=<optimized out>, args=<optimized out>, 
    argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, 
    defcount=0, kwdefs=0x0, closure=0x0) at Python/ceval.c:4149
#87 0x00007ffff7a16f2b in PyEval_EvalCode (co=<optimized out>, 
    globals=<optimized out>, locals=<optimized out>) at Python/ceval.c:695
#88 0x00007ffff7a1496e in builtin_exec_impl (locals=0x7fffed348af8, 
    globals=0x7fffed348af8, source=0x7ffff6677030, module=<optimized out>)
    at Python/bltinmodule.c:974
#89 builtin_exec (module=<optimized out>, args=<optimized out>)
    at Python/clinic/bltinmodule.c.h:281
#90 0x00007ffff7993199 in PyCFunction_Call (func=0x7ffff68569d8, 
    args=0x7fffed5b1c08, kwds=<optimized out>) at Objects/methodobject.c:114
#91 0x00007ffff7a1ecc2 in do_call_core (kwdict=0x7fffed348b88, 
    callargs=<optimized out>, func=0x7ffff68569d8) at Python/ceval.c:5063
#92 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at Python/ceval.c:3366
#93 0x00007ffff7a16a60 in _PyEval_EvalCodeWithName (_co=0x7ffff684ae40, 
    globals=<optimized out>, locals=<optimized out>, args=<optimized out>, 
---Type <return> to continue, or q <return> to quit---
    argcount=3, kwnames=0x0, kwargs=0x7ffff672f938, kwcount=0, kwstep=1, 
    defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x7ffff67f79e0, 
    qualname=0x7ffff67f79e0) at Python/ceval.c:4128
#94 0x00007ffff7a1848a in fast_function (kwnames=<optimized out>, nargs=3, 
    stack=<optimized out>, func=0x7ffff6835ea0) at Python/ceval.c:4939
#95 call_function (pp_stack=0x7fffffffa8d8, oparg=<optimized out>, 
    kwnames=<optimized out>) at Python/ceval.c:4819
#96 0x00007ffff7a1b15d in _PyEval_EvalFrameDefault (f=<optimized out>, 
    throwflag=<optimized out>) at Python/ceval.c:3284
#97 0x00007ffff7a15e74 in _PyFunction_FastCall (co=<optimized out>, 
    args=<optimized out>, nargs=2, globals=<optimized out>)
    at Python/ceval.c:4880
#98 0x00007ffff7a185e8 in fast_function (kwnames=0x0, nargs=2, 
    stack=<optimized out>, func=0x7ffff67bc8c8) at Python/ceval.c:4915
#99 call_function (pp_stack=0x7fffffffab08, oparg=<optimized out>, kwnames=0x0)
    at Python/ceval.c:4819
#100 0x00007ffff7a1b15d in _PyEval_EvalFrameDefault (f=<optimized out>, 
    throwflag=<optimized out>) at Python/ceval.c:3284
#101 0x00007ffff7a15e74 in _PyFunction_FastCall (co=<optimized out>, 
    args=<optimized out>, nargs=1, globals=<optimized out>)
---Type <return> to continue, or q <return> to quit---
    at Python/ceval.c:4880
#102 0x00007ffff7a185e8 in fast_function (kwnames=0x0, nargs=1, 
    stack=<optimized out>, func=0x7ffff6815bf8) at Python/ceval.c:4915
#103 call_function (pp_stack=0x7fffffffad38, oparg=<optimized out>, 
    kwnames=0x0) at Python/ceval.c:4819
#104 0x00007ffff7a1b15d in _PyEval_EvalFrameDefault (f=<optimized out>, 
    throwflag=<optimized out>) at Python/ceval.c:3284
#105 0x00007ffff7a15e74 in _PyFunction_FastCall (co=<optimized out>, 
    args=<optimized out>, nargs=2, globals=<optimized out>)
    at Python/ceval.c:4880
#106 0x00007ffff7a185e8 in fast_function (kwnames=0x0, nargs=2, 
    stack=<optimized out>, func=0x7ffff6818d08) at Python/ceval.c:4915
#107 call_function (pp_stack=0x7fffffffaf68, oparg=<optimized out>, 
    kwnames=0x0) at Python/ceval.c:4819
#108 0x00007ffff7a1b15d in _PyEval_EvalFrameDefault (f=<optimized out>, 
    throwflag=<optimized out>) at Python/ceval.c:3284
#109 0x00007ffff7a15e74 in _PyFunction_FastCall (co=<optimized out>, 
    args=<optimized out>, nargs=2, globals=<optimized out>)
    at Python/ceval.c:4880
#110 0x00007ffff7a16e75 in _PyFunction_FastCallDict (func=0x7ffff6818d90, 
---Type <return> to continue, or q <return> to quit---
    args=0x7fffffffb130, nargs=2, kwargs=0x0) at Python/ceval.c:4982
#111 0x00007ffff793bba6 in _PyObject_FastCallDict (func=0x7ffff6818d90, 
    args=0x7fffffffb130, nargs=<optimized out>, kwargs=0x0)
    at Objects/abstract.c:2295
#112 0x00007ffff793ef4f in _PyObject_CallMethodIdObjArgs (
    callable=0x7ffff6818d90, name=<optimized out>) at Objects/abstract.c:2780
#113 0x00007ffff7a3ac11 in PyImport_ImportModuleLevelObject (
    name=0x7fffed3455b0, globals=<optimized out>, locals=<optimized out>, 
    fromlist=0x0, level=0) at Python/import.c:1592
#114 0x00007ffff7a1529f in builtin___import__ (self=<optimized out>, 
    args=<optimized out>, kwds=<optimized out>) at Python/bltinmodule.c:231
#115 0x00007ffff79931c9 in PyCFunction_Call (func=0x7ffff6856630, 
    args=0x7ffff64189b0, kwds=<optimized out>) at Objects/methodobject.c:98
#116 0x00007ffff7a1ecc2 in do_call_core (kwdict=0x7fffed5b6678, 
    callargs=<optimized out>, func=0x7ffff6856630) at Python/ceval.c:5063
#117 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at Python/ceval.c:3366
#118 0x00007ffff7a16a60 in _PyEval_EvalCodeWithName (_co=0x7ffff684ae40, 
    globals=<optimized out>, locals=<optimized out>, args=<optimized out>, 
    argcount=2, kwnames=0x0, kwargs=0xfa9180, kwcount=0, kwstep=1, defs=0x0, 
---Type <return> to continue, or q <return> to quit---
    defcount=0, kwdefs=0x0, closure=0x0, name=0x7ffff67f79e0, 
    qualname=0x7ffff67f79e0) at Python/ceval.c:4128
#119 0x00007ffff7a1848a in fast_function (kwnames=<optimized out>, nargs=2, 
    stack=<optimized out>, func=0x7ffff6835ea0) at Python/ceval.c:4939
#120 call_function (pp_stack=0x7fffffffb6f8, oparg=<optimized out>, 
    kwnames=<optimized out>) at Python/ceval.c:4819
#121 0x00007ffff7a1b15d in _PyEval_EvalFrameDefault (f=<optimized out>, 
    throwflag=<optimized out>) at Python/ceval.c:3284
#122 0x00007ffff7a15e74 in _PyFunction_FastCall (co=<optimized out>, 
    args=<optimized out>, nargs=3, globals=<optimized out>)
    at Python/ceval.c:4880
#123 0x00007ffff7a16e75 in _PyFunction_FastCallDict (func=0x7ffff6818ea0, 
    args=0x7fffffffb8c0, nargs=3, kwargs=0x0) at Python/ceval.c:4982
#124 0x00007ffff793bba6 in _PyObject_FastCallDict (func=0x7ffff6818ea0, 
    args=0x7fffffffb8c0, nargs=<optimized out>, kwargs=0x0)
    at Objects/abstract.c:2295
#125 0x00007ffff793ef4f in _PyObject_CallMethodIdObjArgs (
    callable=0x7ffff6818ea0, name=<optimized out>) at Objects/abstract.c:2780
#126 0x00007ffff7a3aa46 in PyImport_ImportModuleLevelObject (
    name=0x7ffff6413bb8, globals=<optimized out>, locals=<optimized out>, 
---Type <return> to continue, or q <return> to quit---
    fromlist=0x7ffff6418b00, level=0) at Python/import.c:1658
#127 0x00007ffff7a2046d in import_name (level=0x7ffff7da79a0 <small_ints+160>, 
    fromlist=0x7ffff6418b00, name=0x7ffff6413bb8, f=0x6d5cc8)
    at Python/ceval.c:5176
#128 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at Python/ceval.c:2848
#129 0x00007ffff7a16a60 in _PyEval_EvalCodeWithName (_co=0x7fffed5b7e40, 
    globals=<optimized out>, locals=<optimized out>, args=<optimized out>, 
    argcount=0, kwnames=0x0, kwargs=0x8, kwcount=0, kwstep=2, defs=0x0, 
    defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0)
    at Python/ceval.c:4128
#130 0x00007ffff7a16ee3 in PyEval_EvalCodeEx (_co=<optimized out>, 
    globals=<optimized out>, locals=<optimized out>, args=<optimized out>, 
    argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, 
    defcount=0, kwdefs=0x0, closure=0x0) at Python/ceval.c:4149
#131 0x00007ffff7a16f2b in PyEval_EvalCode (co=<optimized out>, 
    globals=<optimized out>, locals=<optimized out>) at Python/ceval.c:695
#132 0x00007ffff7a1496e in builtin_exec_impl (locals=0x7fffed5ac7e0, 
    globals=0x7fffed5ac7e0, source=0x7fffed5b7e40, module=<optimized out>)
    at Python/bltinmodule.c:974
---Type <return> to continue, or q <return> to quit---
#133 builtin_exec (module=<optimized out>, args=<optimized out>)
    at Python/clinic/bltinmodule.c.h:281
#134 0x00007ffff7993199 in PyCFunction_Call (func=0x7ffff68569d8, 
    args=0x7fffed5b1fc8, kwds=<optimized out>) at Objects/methodobject.c:114
#135 0x00007ffff7a1ecc2 in do_call_core (kwdict=0x7fffed5ac990, 
    callargs=<optimized out>, func=0x7ffff68569d8) at Python/ceval.c:5063
#136 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at Python/ceval.c:3366
#137 0x00007ffff7a16a60 in _PyEval_EvalCodeWithName (_co=0x7ffff684ae40, 
    globals=<optimized out>, locals=<optimized out>, args=<optimized out>, 
    argcount=3, kwnames=0x0, kwargs=0xf783a8, kwcount=0, kwstep=1, defs=0x0, 
    defcount=0, kwdefs=0x0, closure=0x0, name=0x7ffff67f79e0, 
    qualname=0x7ffff67f79e0) at Python/ceval.c:4128
#138 0x00007ffff7a1848a in fast_function (kwnames=<optimized out>, nargs=3, 
    stack=<optimized out>, func=0x7ffff6835ea0) at Python/ceval.c:4939
#139 call_function (pp_stack=0x7fffffffc138, oparg=<optimized out>, 
    kwnames=<optimized out>) at Python/ceval.c:4819
#140 0x00007ffff7a1b15d in _PyEval_EvalFrameDefault (f=<optimized out>, 
    throwflag=<optimized out>) at Python/ceval.c:3284
#141 0x00007ffff7a15e74 in _PyFunction_FastCall (co=<optimized out>, 
---Type <return> to continue, or q <return> to quit---
    args=<optimized out>, nargs=2, globals=<optimized out>)
    at Python/ceval.c:4880
#142 0x00007ffff7a185e8 in fast_function (kwnames=0x0, nargs=2, 
    stack=<optimized out>, func=0x7ffff67bc8c8) at Python/ceval.c:4915
#143 call_function (pp_stack=0x7fffffffc368, oparg=<optimized out>, 
    kwnames=0x0) at Python/ceval.c:4819
#144 0x00007ffff7a1b15d in _PyEval_EvalFrameDefault (f=<optimized out>, 
    throwflag=<optimized out>) at Python/ceval.c:3284
#145 0x00007ffff7a15e74 in _PyFunction_FastCall (co=<optimized out>, 
    args=<optimized out>, nargs=1, globals=<optimized out>)
    at Python/ceval.c:4880
#146 0x00007ffff7a185e8 in fast_function (kwnames=0x0, nargs=1, 
    stack=<optimized out>, func=0x7ffff6815bf8) at Python/ceval.c:4915
#147 call_function (pp_stack=0x7fffffffc598, oparg=<optimized out>, 
    kwnames=0x0) at Python/ceval.c:4819
#148 0x00007ffff7a1b15d in _PyEval_EvalFrameDefault (f=<optimized out>, 
    throwflag=<optimized out>) at Python/ceval.c:3284
#149 0x00007ffff7a15e74 in _PyFunction_FastCall (co=<optimized out>, 
    args=<optimized out>, nargs=2, globals=<optimized out>)
    at Python/ceval.c:4880
---Type <return> to continue, or q <return> to quit---
#150 0x00007ffff7a185e8 in fast_function (kwnames=0x0, nargs=2, 
    stack=<optimized out>, func=0x7ffff6818d08) at Python/ceval.c:4915
#151 call_function (pp_stack=0x7fffffffc7c8, oparg=<optimized out>, 
    kwnames=0x0) at Python/ceval.c:4819
#152 0x00007ffff7a1b15d in _PyEval_EvalFrameDefault (f=<optimized out>, 
    throwflag=<optimized out>) at Python/ceval.c:3284
#153 0x00007ffff7a15e74 in _PyFunction_FastCall (co=<optimized out>, 
    args=<optimized out>, nargs=2, globals=<optimized out>)
    at Python/ceval.c:4880
#154 0x00007ffff7a16e75 in _PyFunction_FastCallDict (func=0x7ffff6818d90, 
    args=0x7fffffffc990, nargs=2, kwargs=0x0) at Python/ceval.c:4982
#155 0x00007ffff793bba6 in _PyObject_FastCallDict (func=0x7ffff6818d90, 
    args=0x7fffffffc990, nargs=<optimized out>, kwargs=0x0)
    at Objects/abstract.c:2295
#156 0x00007ffff793ef4f in _PyObject_CallMethodIdObjArgs (
    callable=0x7ffff6818d90, name=<optimized out>) at Objects/abstract.c:2780
#157 0x00007ffff7a3ac11 in PyImport_ImportModuleLevelObject (
    name=0x7ffff6413bb8, globals=<optimized out>, locals=<optimized out>, 
    fromlist=0x7ffff6418898, level=0) at Python/import.c:1592
#158 0x00007ffff7a2046d in import_name (level=0x7ffff7da79a0 <small_ints+160>, 
---Type <return> to continue, or q <return> to quit---
    fromlist=0x7ffff6418898, name=0x7ffff6413bb8, f=0x6ae3f8)
    at Python/ceval.c:5176
#159 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at Python/ceval.c:2848
#160 0x00007ffff7a16a60 in _PyEval_EvalCodeWithName (_co=0x7ffff6677270, 
    globals=<optimized out>, locals=<optimized out>, args=<optimized out>, 
    argcount=0, kwnames=0x0, kwargs=0x8, kwcount=0, kwstep=2, defs=0x0, 
    defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0)
    at Python/ceval.c:4128
#161 0x00007ffff7a16ee3 in PyEval_EvalCodeEx (_co=<optimized out>, 
    globals=<optimized out>, locals=<optimized out>, args=<optimized out>, 
    argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, 
    defcount=0, kwdefs=0x0, closure=0x0) at Python/ceval.c:4149
#162 0x00007ffff7a16f2b in PyEval_EvalCode (co=<optimized out>, 
    globals=<optimized out>, locals=<optimized out>) at Python/ceval.c:695
#163 0x00007ffff7a1496e in builtin_exec_impl (locals=0x7ffff6413b40, 
    globals=0x7ffff6413b40, source=0x7ffff6677270, module=<optimized out>)
    at Python/bltinmodule.c:974
#164 builtin_exec (module=<optimized out>, args=<optimized out>)
    at Python/clinic/bltinmodule.c.h:281
---Type <return> to continue, or q <return> to quit---
#165 0x00007ffff7993199 in PyCFunction_Call (func=0x7ffff68569d8, 
    args=0x7ffff64089c8, kwds=<optimized out>) at Objects/methodobject.c:114
#166 0x00007ffff7a1ecc2 in do_call_core (kwdict=0x7ffff6413cf0, 
    callargs=<optimized out>, func=0x7ffff68569d8) at Python/ceval.c:5063
#167 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at Python/ceval.c:3366
#168 0x00007ffff7a16a60 in _PyEval_EvalCodeWithName (_co=0x7ffff684ae40, 
    globals=<optimized out>, locals=<optimized out>, args=<optimized out>, 
    argcount=3, kwnames=0x0, kwargs=0x7fffed5d71f8, kwcount=0, kwstep=1, 
    defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x7ffff67f79e0, 
    qualname=0x7ffff67f79e0) at Python/ceval.c:4128
#169 0x00007ffff7a1848a in fast_function (kwnames=<optimized out>, nargs=3, 
    stack=<optimized out>, func=0x7ffff6835ea0) at Python/ceval.c:4939
#170 call_function (pp_stack=0x7fffffffd208, oparg=<optimized out>, 
    kwnames=<optimized out>) at Python/ceval.c:4819
#171 0x00007ffff7a1b15d in _PyEval_EvalFrameDefault (f=<optimized out>, 
    throwflag=<optimized out>) at Python/ceval.c:3284
#172 0x00007ffff7a15e74 in _PyFunction_FastCall (co=<optimized out>, 
    args=<optimized out>, nargs=2, globals=<optimized out>)
    at Python/ceval.c:4880
---Type <return> to continue, or q <return> to quit---
#173 0x00007ffff7a185e8 in fast_function (kwnames=0x0, nargs=2, 
    stack=<optimized out>, func=0x7ffff67bc8c8) at Python/ceval.c:4915
#174 call_function (pp_stack=0x7fffffffd438, oparg=<optimized out>, 
    kwnames=0x0) at Python/ceval.c:4819
#175 0x00007ffff7a1b15d in _PyEval_EvalFrameDefault (f=<optimized out>, 
    throwflag=<optimized out>) at Python/ceval.c:3284
#176 0x00007ffff7a15e74 in _PyFunction_FastCall (co=<optimized out>, 
    args=<optimized out>, nargs=1, globals=<optimized out>)
    at Python/ceval.c:4880
#177 0x00007ffff7a185e8 in fast_function (kwnames=0x0, nargs=1, 
    stack=<optimized out>, func=0x7ffff6815bf8) at Python/ceval.c:4915
#178 call_function (pp_stack=0x7fffffffd668, oparg=<optimized out>, 
    kwnames=0x0) at Python/ceval.c:4819
#179 0x00007ffff7a1b15d in _PyEval_EvalFrameDefault (f=<optimized out>, 
    throwflag=<optimized out>) at Python/ceval.c:3284
#180 0x00007ffff7a15e74 in _PyFunction_FastCall (co=<optimized out>, 
    args=<optimized out>, nargs=2, globals=<optimized out>)
    at Python/ceval.c:4880
#181 0x00007ffff7a185e8 in fast_function (kwnames=0x0, nargs=2, 
    stack=<optimized out>, func=0x7ffff6818d08) at Python/ceval.c:4915
---Type <return> to continue, or q <return> to quit---
#182 call_function (pp_stack=0x7fffffffd898, oparg=<optimized out>, 
    kwnames=0x0) at Python/ceval.c:4819
#183 0x00007ffff7a1b15d in _PyEval_EvalFrameDefault (f=<optimized out>, 
    throwflag=<optimized out>) at Python/ceval.c:3284
#184 0x00007ffff7a15e74 in _PyFunction_FastCall (co=<optimized out>, 
    args=<optimized out>, nargs=2, globals=<optimized out>)
    at Python/ceval.c:4880
#185 0x00007ffff7a16e75 in _PyFunction_FastCallDict (func=0x7ffff6818d90, 
    args=0x7fffffffda60, nargs=2, kwargs=0x0) at Python/ceval.c:4982
#186 0x00007ffff793bba6 in _PyObject_FastCallDict (func=0x7ffff6818d90, 
    args=0x7fffffffda60, nargs=<optimized out>, kwargs=0x0)
    at Objects/abstract.c:2295
#187 0x00007ffff793ef4f in _PyObject_CallMethodIdObjArgs (
    callable=0x7ffff6818d90, name=<optimized out>) at Objects/abstract.c:2780
#188 0x00007ffff7a3ac11 in PyImport_ImportModuleLevelObject (
    name=0x7fffed5c39f0, globals=<optimized out>, locals=<optimized out>, 
    fromlist=0x7ffff7d5dcd0 <_Py_NoneStruct>, level=0) at Python/import.c:1592
#189 0x00007ffff7a2046d in import_name (level=0x7ffff7da79a0 <small_ints+160>, 
    fromlist=0x7ffff7d5dcd0 <_Py_NoneStruct>, name=0x7fffed5c39f0, 
    f=0x7ffff6734828) at Python/ceval.c:5176
---Type <return> to continue, or q <return> to quit---
#190 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at Python/ceval.c:2848
#191 0x00007ffff7a16a60 in _PyEval_EvalCodeWithName (_co=0x7ffff666a930, 
    globals=<optimized out>, locals=<optimized out>, args=<optimized out>, 
    argcount=0, kwnames=0x0, kwargs=0x8, kwcount=0, kwstep=2, defs=0x0, 
    defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0)
    at Python/ceval.c:4128
#192 0x00007ffff7a16ee3 in PyEval_EvalCodeEx (_co=<optimized out>, 
    globals=<optimized out>, locals=<optimized out>, args=<optimized out>, 
    argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, 
    defcount=0, kwdefs=0x0, closure=0x0) at Python/ceval.c:4149
#193 0x00007ffff7a16f2b in PyEval_EvalCode (co=<optimized out>, 
    globals=<optimized out>, locals=<optimized out>) at Python/ceval.c:695
#194 0x00007ffff7a4b8d6 in run_mod (arena=0x7ffff67f1348, 
    flags=0x7fffffffe070, locals=0x7ffff67d4f30, globals=0x7ffff67d4f30, 
    filename=0x7ffff678d8f0, mod=0x6d27d8) at Python/pythonrun.c:980
#195 PyRun_InteractiveOneObject (fp=<optimized out>, filename=0x7ffff678d8f0, 
    flags=0x7fffffffe070) at Python/pythonrun.c:233
#196 0x00007ffff7a4bc3e in PyRun_InteractiveLoopFlags (
    fp=0x7ffff6d958e0 <_IO_2_1_stdin_>, filename_str=<optimized out>, 
---Type <return> to continue, or q <return> to quit---
    flags=0x7fffffffe070) at Python/pythonrun.c:112
#197 0x00007ffff7a4bd7c in PyRun_AnyFileExFlags (
    fp=0x7ffff6d958e0 <_IO_2_1_stdin_>, filename=0x7ffff7ae670b "<stdin>", 
    closeit=0, flags=0x7fffffffe070) at Python/pythonrun.c:74
#198 0x00007ffff7a6629b in run_file (p_cf=0x7fffffffe070, filename=0x0, 
    fp=0x7ffff6d958e0 <_IO_2_1_stdin_>) at Modules/main.c:338
#199 Py_Main (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:810
#200 0x0000000000400c1d in main (argc=1, argv=<optimized out>)
    at ./Programs/python.c:69
(gdb) 

[robertnishihara]

Note that the segfault happens if we replace import ray with import pyarrow.

cc @pcmoritz

[pcmoritz]

Could it be related to the change in symbols.map here: apache/arrow@2093f6e?

[pcmoritz]

If not, doing a search on pyarrow commits and seeing where it was introduced will be helpful

[robertnishihara]

Unfortunately when pyarrow is compiled from source the problem disappears, so it has something to do with the way wheels are built.

[robertnishihara]

Tracked it down a bit further in https://issues.apache.org/jira/browse/ARROW-2657.

[DaveyBiggers]

Note that this seems to make it impossible to use actors which have a dependency on tensorflow. Even if the imports are ordered correctly, when ray unpickles the actor class it will import tensorflow automatically, causing the agent to die.

Here's a minimal example:

# test.py
import tensorflow as tf
import ray
import time

@ray.remote
class AgentWithTensorFlowDependency:
    def __init__(self):
        print("Actor: Tensorflow version:",tf.VERSION)

def main():
    ray.init(redirect_output=True)
    print("Main: Tensorflow version:",tf.VERSION)
    agent = AgentWithTensorFlowDependency.remote()
    time.sleep(10)

if __name__ == '__main__':
    main()

Running python3 test.py produces the following for me:

Main: Tensorflow version: 1.8.0
The worker with ID 56a7dcfae144fb6d6e1f193920edba5042e28833 died or was killed while executing the task with ID 0108b664879dbd96c8b9e30b1419943206ef2b28
The worker with ID d281d7af5e1edea3a5de1f598382b8b32e45b037 died or was killed while executing the task with ID 0108b664879dbd96c8b9e30b1419943206ef2b28
The worker with ID d2aebc63b6750c7ef24bdcf4680b617588c4bbf3 died or was killed while executing the task with ID 0108b664879dbd96c8b9e30b1419943206ef2b28
The worker with ID 3c69720cb568fcb6bb712c4faa708bc1f3c247cd died or was killed while executing the task with ID 0108b664879dbd96c8b9e30b1419943206ef2b28
The worker with ID bb400eef52783d1289fad51d85964037df18e1da died or was killed while executing the task with ID 0108b664879dbd96c8b9e30b1419943206ef2b28

[Scitator]

@robertnishihara @DaveyBiggers
Does anyone know a hotfix to avoid this problem for a wild?

[robertnishihara]

Compiling from source should work. Also importing tensorflow before ray should work (though that isn't possible on ray workers). Or installing Ray from a commit from before 225608e should work as well.