-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ARROW-2657: [Python] Import TensorFlow python extension before pyarrow to avoid segfault #2210
Changes from 6 commits
1c9628f
7835fba
02cb500
57ca5fc
1135b51
ac38837
c18cccb
bbf6cfc
70f3bca
2ca3de9
92aef7a
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -44,6 +44,25 @@ def parse_version(root): | |
__version__ = None | ||
|
||
|
||
try: | ||
# We need to do the following to load TensorFlow before | ||
# pyarrow.lib. If we don't do this there are symbol clashes | ||
# between TensorFlow's use of threading and our global | ||
# thread pool, see also | ||
# https://issues.apache.org/jira/browse/ARROW-2657 and | ||
# https://github.com/apache/arrow/pull/2096. | ||
import ctypes | ||
import os | ||
from sys import platform | ||
import site | ||
if platform == "linux" or platform == "linux2": | ||
SITE_PATH, = site.getsitepackages() | ||
ctypes.CDLL(os.path.join(SITE_PATH, "tensorflow", | ||
"libtensorflow_framework.so")) | ||
except: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It might be better to make this less restrictive. You could even get away without a try/except and only attempt to load the shared lib if the TF folder is there. How about putting this in a function in pyarrow.compat? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. fixed |
||
pass | ||
|
||
|
||
from pyarrow.lib import cpu_count, set_cpu_count | ||
from pyarrow.lib import (null, bool_, | ||
int8, int16, int32, int64, | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did this test fail before this fix?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I tried it on an ec2 instance before putting it in.