PyThreadState Sometimes Used for Different Threads for Subinterpreters #109860
Assignees
Labels
3.12
bugs and security fixes
3.13
bugs and security fixes
interpreter-core
(Objects, Python, Grammar, and Parser dirs)
topic-subinterpreters
type-bug
An unexpected behavior, bug, or error
Comments
ericsnowcurrently
added
type-bug
ericsnowcurrently
added a commit
that referenced
this issue
Oct 3, 2023
…ecessary (gh-110245) In a few places we switch to another interpreter without knowing if it has a thread state associated with the current thread. For the main interpreter there wasn't much of a problem, but for subinterpreters we were *mostly* okay re-using the tstate created with the interpreter (located via PyInterpreterState_ThreadHead()). There was a good chance that tstate wasn't actually in use by another thread. However, there are no guarantees of that. Furthermore, re-using an already used tstate is currently fragile. To address this, now we create a new thread state in each of those places and use it. One consequence of this change is that PyInterpreterState_ThreadHead() may not return NULL (though that won't happen for the main interpreter).
ericsnowcurrently
added a commit
to ericsnowcurrently/cpython
that referenced
this issue
Oct 11, 2023
…eters, When Necessary (pythongh-110245) In a few places we switch to another interpreter without knowing if it has a thread state associated with the current thread. For the main interpreter there wasn't much of a problem, but for subinterpreters we were *mostly* okay re-using the tstate created with the interpreter (located via PyInterpreterState_ThreadHead()). There was a good chance that tstate wasn't actually in use by another thread. However, there are no guarantees of that. Furthermore, re-using an already used tstate is currently fragile. To address this, now we create a new thread state in each of those places and use it. One consequence of this change is that PyInterpreterState_ThreadHead() may not return NULL (though that won't happen for the main interpreter). (cherry-picked from commit f5198b0)
ericsnowcurrently
added a commit
to ericsnowcurrently/cpython
that referenced
this issue
Oct 12, 2023
…eters, When Necessary (pythongh-110245) In a few places we switch to another interpreter without knowing if it has a thread state associated with the current thread. For the main interpreter there wasn't much of a problem, but for subinterpreters we were *mostly* okay re-using the tstate created with the interpreter (located via PyInterpreterState_ThreadHead()). There was a good chance that tstate wasn't actually in use by another thread. However, there are no guarantees of that. Furthermore, re-using an already used tstate is currently fragile. To address this, now we create a new thread state in each of those places and use it. One consequence of this change is that PyInterpreterState_ThreadHead() may not return NULL (though that won't happen for the main interpreter). (cherry-picked from commit f5198b0)
Glyphack
pushed a commit
to Glyphack/cpython
that referenced
this issue
Sep 2, 2024
…When Necessary (pythongh-110245) In a few places we switch to another interpreter without knowing if it has a thread state associated with the current thread. For the main interpreter there wasn't much of a problem, but for subinterpreters we were *mostly* okay re-using the tstate created with the interpreter (located via PyInterpreterState_ThreadHead()). There was a good chance that tstate wasn't actually in use by another thread. However, there are no guarantees of that. Furthermore, re-using an already used tstate is currently fragile. To address this, now we create a new thread state in each of those places and use it. One consequence of this change is that PyInterpreterState_ThreadHead() may not return NULL (though that won't happen for the main interpreter).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Bug report
In a few places, we treat
PyInterpreterState.threads.head(AKAPyInterpreterState_ThreadHead()) for subinterpreters as an unused thread state that we can use in a one-off manner:_PyInterpreterState_IDDecref()(Python/pystate.c)_call_in_interpreter()(Python/pystate.c) - fixed in 3.13+ (gh-76785: Use Pending Calls When Releasing Cross-Interpreter Data #109556)_run_script_in_interpreter()(Modules/_xxsubinterpreter.c)interp_destroy()(Modules/_xxsubinterpreter.c)The problem is that each thread state should correspond to a single OS thread, and for each OS thread there should be at most one thread state per interpreter. Using
PyInterpreterState.threads.headlike that violates those assumptions, which can cause problems. 1Also, some code assumes
PyInterpreterState.threads.headis the current thread state, when it might not be:interpreter_clear()(Python/pystate.c)We should fix this by always using the thread state that corresponds to the current thread (and create one if missing). For efficiency, we can use a thread-local (via
PyThread_tss_*) to store each interpreter's thread state for that thread.Linked PRs
Footnotes
Currently, this mostly doesn't affect the main interpreter. Ideally, the main interpreter wouldn't actually be treated specially. See (Mostly) Stop Special-casing the Main Interpreter #109857. ↩
The text was updated successfully, but these errors were encountered: