-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ARROW-2657: [Python] Import TensorFlow python extension before pyarrow to avoid segfault #2210
Conversation
This is now ready for review! This is only tested on linux, where are the mac wheels being built? |
Codecov Report
@@ Coverage Diff @@
## master #2210 +/- ##
==========================================
- Coverage 86.52% 84.07% -2.45%
==========================================
Files 244 288 +44
Lines 41893 43771 +1878
==========================================
+ Hits 36246 36801 +555
- Misses 5647 6939 +1292
- Partials 0 31 +31
Continue to review full report at Codecov.
|
python/pyarrow/__init__.py
Outdated
SITE_PATH, = site.getsitepackages() | ||
ctypes.CDLL(os.path.join(SITE_PATH, "tensorflow", | ||
"libtensorflow_framework.so")) | ||
except: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It might be better to make this less restrictive. You could even get away without a try/except and only attempt to load the shared lib if the TF folder is there. How about putting this in a function in pyarrow.compat?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed
|
||
# Testing for https://issues.apache.org/jira/browse/ARROW-2657 | ||
# These tests cannot be run inside of the docker container, since TensorFlow | ||
# does not run on manylinux1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did this test fail before this fix?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I tried it on an ec2 instance before putting it in.
Do you anticipate this fix breaking in any cases? |
@robertnishihara So I think the current version should be pretty robust and cover the corner cases (there might be some corner cases where it might incur the full TensorFlow loading time, but in most cases it will be fast) |
This should now be ready to merge, let me know if there are other comments! I plan to merge it later tonight/early tomorrow morning if there are none. |
Yeah, the primary failure mode that I'm thinking of is somebody modifies |
python/pyarrow/__init__.py
Outdated
|
||
|
||
# Workaround for https://issues.apache.org/jira/browse/ARROW-2657 | ||
import_tensorflow_extension() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: this exposes import_tensorflow_extension
in the pyarrow.* namespace. I'll fix
…yarrow.* namespace Change-Id: I94bd60d98b64cab6105fbdda793bc82050dc640e
+1. Will merge once the builds complete |
|
||
|
||
# Workaround for https://issues.apache.org/jira/browse/ARROW-2657 | ||
compat.import_tensorflow_extension() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn't this be run only on Linux?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, sorry I missed that (it was in the prior iteration of the patch but I didn't look closely enough at this function): https://issues.apache.org/jira/browse/ARROW-2795
…latforms Per comments in #2210 Author: Wes McKinney <[email protected]> Closes #2218 from wesm/ARROW-2795 and squashes the following commits: 9825fbf <Wes McKinney> Run TensorFlow import workaround only on Linux platforms
No description provided.