[Python] Segmentation fault when pyarrow is imported in exit handler #38626
Assignees
Labels
Component: Python
Critical Fix
Bugfixes for security vulnerabilities, crashes, or invalid data.
Type: bug
Milestone
Comments
kou
changed the title
|
Why do you want to do this...? |
|
Exit handlers in a complex codebase might incidentally import pyarrow especially via pandas. |
|
I can confirm the segfault, both with pyarrow 13.0 and main from a few days ago. The gdb bactrace: |
|
General cpython issue about issues with threads in finalization: python/cpython#80657 (cc @pitrou) We do have some protection around acquiring the GIL here, but so that is not sufficient: arrow/python/pyarrow/src/arrow/python/common.h Lines 227 to 235 in e62ec62 |
|
@jorisvandenbossche Is it the full backtrace? Are there other threads running? |
|
Ok, here is the full backtrace: |
pitrou
changed the title
pitrou
added a commit
to pitrou/arrow
that referenced
this issue
Nov 8, 2023
…tdown Some C++ destructors may be called after the Python interpreter has ceased to exist.
pitrou
added a commit
to pitrou/arrow
that referenced
this issue
Nov 8, 2023
…tdown Some C++ destructors may be called after the Python interpreter has ceased to exist.
jorisvandenbossche
pushed a commit
that referenced
this issue
Nov 14, 2023
…38637) ### Rationale for this change Some C++ destructors may be called after the Python interpreter has ceased to exist. If such a destructor tries to call back in the Python interpreter, for example by calling `Py_DECREF`, we get a crash. ### What changes are included in this PR? Protect `OwnedRef` and `OwneRefNoGIL` destructors against decref'ing a Python object after Python finalization. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * Closes: #38626 Authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Joris Van den Bossche <[email protected]>
raulcd
pushed a commit
that referenced
this issue
Nov 28, 2023
…38637) ### Rationale for this change Some C++ destructors may be called after the Python interpreter has ceased to exist. If such a destructor tries to call back in the Python interpreter, for example by calling `Py_DECREF`, we get a crash. ### What changes are included in this PR? Protect `OwnedRef` and `OwneRefNoGIL` destructors against decref'ing a Python object after Python finalization. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * Closes: #38626 Authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Joris Van den Bossche <[email protected]>
amoeba
added
the
Critical Fix
dgreiss
pushed a commit
to dgreiss/arrow
that referenced
this issue
Feb 19, 2024
…tdown (apache#38637) ### Rationale for this change Some C++ destructors may be called after the Python interpreter has ceased to exist. If such a destructor tries to call back in the Python interpreter, for example by calling `Py_DECREF`, we get a crash. ### What changes are included in this PR? Protect `OwnedRef` and `OwneRefNoGIL` destructors against decref'ing a Python object after Python finalization. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * Closes: apache#38626 Authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Joris Van den Bossche <[email protected]>
|
please, I am having problems, I have tried many time find a solution and failed miserably:
|
|
@asda10 This does not seem related to this issue, can you open a new issue for it? Also please include a reproducer so that we can try it out ourselves. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug, including details regarding any error messages, version, and platform.
Versions
Component(s)
Python
The text was updated successfully, but these errors were encountered: