Skip to content

99 Old Notes

Jump to bottom
Eric Snow edited this page Sep 8, 2018 · 1 revision
  • "interpreters" API
  • API resources
  • C-API
  • interp state
  • globals
  • PEP 432
  • API examples
  • feedback
  • resources

"interpreters" API

"_interpreter" module (low-level)

(inspired by _threading)

  • get_stack_size() -> int
  • set_stack_size(size)
  • enumerate() -> IDs?
  • current() -> ID?
  • get_info(id)
  • create() -> ID?
  • run(id, target, args=(), kwargs=None)
  • destroy(id)

"interpreter" module (high-level)

(inspired by threading)

  • active_count() -> int
  • current_subinterpreter() -> Interpreter
  • get_ident() -> ID
  • enumerate() -> list of Interpreter
  • main_interpreter() -> Interpreter
  • settrace(func)
  • running -> SubinterpreterExecution
  • stack_size(size=None)

class Interpreter()

  • (classmethod) from_id(cls, id)
  • ident -> string
  • exitcode -> int ???
  • running -> bool
  • is_alive() -> bool
  • run(func_or_code, args=(), kwargs=None, name=None) -> SubinterpreterExecution
  • join(timeout=None)
  • destroy()

class SubinterpreterExecution(group=None, target=None, name=None, args=(), kwargs=None, *, daemon=None)

  • name -> str
  • daemon -> bool
  • start()
  • run()
  • join(timeout=None)
  • is_alive()
  • exitcode()

channels

???

shared-nothing "thread" concurrency model

???

  • built on top of Interpreter?

C-API

  • interpreter (Py_InterpreterState, Py_ThreadState)
  • process-global list of (main) interpreters

...

  • int PyInterpreter_Current_ID()
  • int[] PyInterpreter_Get(char* name)

...

  • mode

...

  • associate with interpreter?
  • PyObject.ob_interp (*PyInterpreterState)

API resources

prior art for interpreter management

(https://docs.python.org/3.5/c-api/init.html)

  • "Py_Initialize()
    • Ex()"
  • Py_IsInitialized()
  • Py_Finalize()
  • PyInterpreterState (see below)
  • PyInterpreterState_New()
  • PyInterpreterState_Clear()
  • PyInterpreterState_Delete()
  • Py_NewInterpreter()
  • Py_EndInterpreter()
  • PyInterpreterState_Head()
  • PyInterpreterState_Next()
  • PyInterpreterState_ThreadHead()

prior art for thread management

(https://docs.python.org/3.5/c-api/init.html)

  • PyThreadState (see below)
  • PyEval_InitThreads()
  • PyEval_ThreadsInitialized()
  • PyEval_SaveThread()
  • PyEval_RestoreThread()
  • PyThreadState_Get()
  • PyThreadState_Swap()
  • PyEval_ReInitThreads()
  • PyGILState_Ensure()
  • PyGILState_Release()
  • PyGILState_GetThisThreadState()
  • PyGILState_Check()
  • PyThreadState_New()
  • PyThreadState_Clear()
  • PyThreadState_Delete()
  • PyThreadState_GetDict()
  • PyThreadState_SetAsyncExc()
  • PyEval_AcquireThread()
  • PyEval_ReleaseThread()
  • PyEval_AcquireLock()
  • PyEval_ReleaseLock()
  • PyThreadState_Next()

prior art for "run"

(https://docs.python.org/3/library/runpy.html)

  • runpy.run_module(mod_name, init_globals=None, run_name=None, alter_sys=False)
  • runpy.run_path(path_name, init_globals=None, run_name=None)
  • runpy._run_module_as_main(mod_name, alter_argv=True)
  • runpy._run_module_code(code, init_globals=None, mod_name=None, mod_spec=None, pkg_name=None, script_name=None)
  • runpy._run_code(code, run_globals, init_globals=None, mod_name=None, mod_spec=None, pkg_name=None, script_name=None)

(https://docs.python.org/3.5/c-api/veryhigh.html)

  • "PyRun_AnyFile()
    • Flags()
    • Ex()
    • ExFlags()"
  • "PyRun_SimpleString()
    • Flags()"
  • "PyRun_SimpleFile()
    • Ex()
    • ExFlags()"
  • "PyRun_InteractiveOne()
    • Flags()"
  • "PyRun_InteractiveLoop()
    • Flags()"
  • "PyRun_String()
    • Flags()"
  • "PyRun_File()
    • Flags()
    • Ex()
    • ExFlags()"

C-API

memory management

(https://docs.python.org/3.5/c-api/memory.html)

reference counting

(https://docs.python.org/3.5/c-api/refcounting.html) (https://docs.python.org/3.5/c-api/gcsupport.html) (https://docs.python.org/3.5/c-api/intro.html#objects-types-and-reference-counts)

import system

(https://docs.python.org/3.5/c-api/import.html) (https://docs.python.org/3.5/c-api/module.html)

  • PyModule_GetState()
  • PyModule_GetDef()
  • PyModuleDef
  • PyModule_Create()
  • PyModule_Create2()
  • PyModuleDef_Init()
  • PyModuleDef_Slot
  • PyModule_FromDefAndSpec()
  • PyModule_FromDefAndSpec2()
  • PyModule_ExecDef()

other

key types

interp state

PyInterpreterState fields

PyThreadState fields

globals

  • static globals (including PyAPI_DATA) (use "nm" command)
  • haven't check static locals...

interpreter/eval state

(https://hg.python.org/cpython/file/default/Include/pystate.h)

  • _Py_atomic_address _PyThreadState_Current
  • PyThreadFrameGetter _PyThreadState_GetFrame

(https://hg.python.org/cpython/file/default/Include/pylifecycle.h)

  • PyThreadState *_Py_Finalizing

(https://hg.python.org/cpython/file/default/Include/pythonrun.h)

  • int (*PyOS_InputHook)(void)
  • char *(*PyOS_ReadlineFunctionPointer)(FILE *, FILE *, const char *)
  • PyThreadState* _PyOS_ReadlineTState

(https://hg.python.org/cpython/file/default/Include/ceval.h)

  • int _Py_CheckRecursionLimit

(https://hg.python.org/cpython/file/default/Python/ceval_gil.h)

  • unsigned long gil_interval
  • _Py_atomic_int gil_locked
  • unsigned long gil_switch_number
  • _Py_atomic_address gil_last_holder
  • COND_T gil_cond
  • MUTEX_T gil_mutex
  • COND_T switch_cond
  • MUTEX_T switch_mutex

(https://hg.python.org/cpython/file/default/Include/pydebug.h) int Py_DebugFlag

  • int Py_VerboseFlag
  • int Py_QuietFlag
  • int Py_InteractiveFlag
  • int Py_InspectFlag
  • int Py_OptimizeFlag
  • int Py_NoSiteFlag
  • int Py_BytesWarningFlag
  • int Py_UseClassExceptionsFlag
  • int Py_FrozenFlag
  • int Py_IgnoreEnvironmentFlag
  • int Py_DontWriteBytecodeFlag
  • int Py_NoUserSiteDirectory
  • int Py_UnbufferedStdioFlag
  • int Py_HashRandomizationFlag
  • int Py_IsolatedFlag

(https://hg.python.org/cpython/file/default/Include/traceback.h)

  • void _Py_DumpTraceback(int fd, PyThreadState *tstate)
  • const char* _Py_DumpTracebackThreads(int fd, PyInterpreterState *interp, PyThreadState *current_thread)

(https://hg.python.org/cpython/file/default/Python/pystate.c)

  • PyThread_type_lock head_mutex
  • PyInterpreterState *autoInterpreterState
  • int autoTLSkey
  • PyInterpreterState *interp_head

(https://hg.python.org/cpython/file/default/Python/pylifecycle.c)

  • int initialized
  • char *_Py_StandardStreamEncoding
  • char *_Py_StandardStreamErrors
  • wchar_t *progname
  • wchar_t *default_home
  • wchar_t env_home[]
  • void (*exitfuncs[])(void)
  • int nexitfuncs

(https://hg.python.org/cpython/file/default/Python/pythonrun.c)

  • _Py_static_string(PyId_string, "")

(https://hg.python.org/cpython/file/default/Python/ceval.c)

  • int lltrace
  • long dxpairs[][]
  • long dxp[]
  • int pcall[]
  • PyThread_type_lock pending_lock
  • long main_thread
  • _Py_atomic_int eval_breaker
  • _Py_atomic_int gil_drop_request
  • _Py_atomic_int pendingcalls_to_do
  • int pending_async_exc
  • struct {...} pendingcalls[]
  • int pendingfirst
  • int pendinglast
  • _Py_atomic_int pendingcalls_to_do
  • int recursion_limit
  • int _Py_CheckRecursionLimit
  • int _Py_TracingPossible

(https://hg.python.org/cpython/file/default/Python/sysmodule.c)

  • PyObject *whatstrings[]
  • int _check_interval
  • PyObject *xoptions

threading

(https://hg.python.org/cpython/file/default/Python/thread.c)

  • int thread_debug
  • int initialized
  • size_t _pythread_stacksize
  • struct key *keyhead
  • PyThread_type_lock keymutex
  • int nkeys

type system

(https://hg.python.org/cpython/file/default/Include/object.h)

  • Py_ssize_t _Py_RefTotal
  • PyObject _Py_NoneStruct
  • PyObject _Py_NotImplementedStruct
  • int _Py_SwappedOp[]

compiler

(https://hg.python.org/cpython/file/default/Include/token.h)

  • const char * _PyParser_TokenNames[]

(https://hg.python.org/cpython/file/default/Python/Python-ast.c)

  • char *Module_fields[]
  • char *Interactive_fields[]
  • char *Expression_fields[]
  • char *Suite_fields[]
  • char *stmt_attributes[]
  • char *FunctionDef_fields[]
  • char *AsyncFunctionDef_fields[]
  • char *ClassDef_fields[]
  • char *Return_fields[]
  • char *Delete_fields[]
  • char *Assign_fields[]
  • char *AugAssign_fields[]
  • char *For_fields[]
  • char *AsyncFor_fields[]
  • char *While_fields[]
  • char *If_fields[]
  • char *With_fields[]
  • char *AsyncWith_fields[]
  • char *Raise_fields[]
  • char *Try_fields[]
  • char *Assert_fields[]
  • char *Import_fields[]
  • char *ImportFrom_fields[]
  • char *Global_fields[]
  • char *Nonlocal_fields[]
  • char *Expr_fields[]
  • char *expr_attributes[]
  • char *BoolOp_fields[]
  • char *BinOp_fields[]
  • char *UnaryOp_fields[]
  • char *Lambda_fields[]
  • char *IfExp_fields[]
  • char *Dict_fields[]
  • char *Set_fields[]
  • char *ListComp_fields[]
  • char *SetComp_fields[]
  • char *DictComp_fields[]
  • char *GeneratorExp_fields[]
  • char *Await_fields[]
  • char *Yield_fields[]
  • char *YieldFrom_fields[]
  • char *Compare_fields[]
  • char *Call_fields[]
  • char *Num_fields[]
  • char *Str_fields[]
  • char *Bytes_fields[]
  • char *NameConstant_fields[]
  • char *Attribute_fields[]
  • char *Subscript_fields[]
  • char *Starred_fields[]
  • char *Name_fields[]
  • char *List_fields[]
  • char *Tuple_fields[]
  • char *Slice_fields[]
  • char *ExtSlice_fields[]
  • char *Index_fields[]
  • char *comprehension_fields[]
  • char *excepthandler_attributes[]
  • char *ExceptHandler_fields[]
  • char *arguments_fields[]
  • char *arg_attributes[]
  • char *arg_fields[]
  • char *keyword_fields[]
  • char *alias_fields[]
  • char *withitem_fields[]

(https://hg.python.org/cpython/file/default/Python/compile.c)

  • PyObject *__doc__

import system

(https://hg.python.org/cpython/file/default/Include/import.h)

  • struct _inittab * PyImport_Inittab
  • const struct _frozen * PyImport_FrozenModules

(https://hg.python.org/cpython/file/default/Include/modsupport.h)

  • char * _Py_PackageContext

memory management

...

builtin types

(https://hg.python.org/cpython/file/default/Include/datetime.h)

  • PyDateTime_CAPI *PyDateTimeAPI

(https://hg.python.org/cpython/file/default/Include/fileobject.h)

  • const char * Py_FileSystemDefaultEncoding
  • int Py_HasFileSystemDefaultEncoding

(https://hg.python.org/cpython/file/default/Include/longobject.h)

  • unsigned char _PyLong_DigitValue[]

(https://hg.python.org/cpython/file/default/Include/unicodeobject.h)

  • const unsigned char _Py_ascii_whitespace[]

(https://hg.python.org/cpython/file/default/Python/_warnings.c)

  • PyObject *_filters
  • PyObject *_once_registry
  • PyObject *_default_action
  • long _filters_version

(https://hg.python.org/cpython/file/default/Python/ast.c)

  • const char* FORBIDDEN[]

(https://hg.python.org/cpython/file/default/Python/codecs.c)

  • _PyUnicode_Name_CAPI *ucnhash_CAPI

(https://hg.python.org/cpython/file/default/Python/dtoa.c)

  • double private_mem[]
  • double *pmem_next
  • Bigint *freelist[]
  • Bigint *p5s

os

(https://hg.python.org/cpython/file/default/Include/py_curses.h)

  • void **PyCurses_API
  • char *catchall_ERR
  • char *catchall_NULL

(https://hg.python.org/cpython/file/default/Include/pygetopt.h)

  • int _PyOS_opterr
  • int _PyOS_optind
  • wchar_t * _PyOS_optarg

helpers

(https://hg.python.org/cpython/file/default/Include/pyctype.h)

  • const unsigned int _Py_ctype_table[]
  • const unsigned char _Py_ctype_tolower[]
  • const unsigned char _Py_ctype_toupper[]

(https://hg.python.org/cpython/file/default/Include/pyhash.h)

  • _Py_HashSecret_t _Py_HashSecret
  • int _Py_HashSecret_Initialized

(https://hg.python.org/cpython/file/default/Include/pymath.h)

  • union {...} __nan_store

singletons

(https://hg.python.org/cpython/file/default/Include/boolobject.h)

  • struct _longobject _Py_FalseStruct
  • struct _longobject _Py_TrueStruct

(https://hg.python.org/cpython/file/default/Include/bytearrayobject.h)

  • char _PyByteArray_empty_string[]

(https://hg.python.org/cpython/file/default/Include/codecs.h)

  • const char * Py_hexdigits

(https://hg.python.org/cpython/file/default/Include/pyerrors.h)

  • PyObject * PyExc_BaseException

  • PyObject * PyExc_Exception

  • PyObject * PyExc_StopAsyncIteration

  • PyObject * PyExc_StopIteration

  • PyObject * PyExc_GeneratorExit

  • PyObject * PyExc_ArithmeticError

  • PyObject * PyExc_LookupError

  • PyObject * PyExc_AssertionError

  • PyObject * PyExc_AttributeError

  • PyObject * PyExc_BufferError

  • PyObject * PyExc_EOFError

  • PyObject * PyExc_FloatingPointError

  • PyObject * PyExc_OSError

  • PyObject * PyExc_ImportError

  • PyObject * PyExc_IndexError

  • PyObject * PyExc_KeyError

  • PyObject * PyExc_KeyboardInterrupt

  • PyObject * PyExc_MemoryError

  • PyObject * PyExc_NameError

  • PyObject * PyExc_OverflowError

  • PyObject * PyExc_RuntimeError

  • PyObject * PyExc_RecursionError

  • PyObject * PyExc_NotImplementedError

  • PyObject * PyExc_SyntaxError

  • PyObject * PyExc_IndentationError

  • PyObject * PyExc_TabError

  • PyObject * PyExc_ReferenceError

  • PyObject * PyExc_SystemError

  • PyObject * PyExc_SystemExit

  • PyObject * PyExc_TypeError

  • PyObject * PyExc_UnboundLocalError

  • PyObject * PyExc_UnicodeError

  • PyObject * PyExc_UnicodeEncodeError

  • PyObject * PyExc_UnicodeDecodeError

  • PyObject * PyExc_UnicodeTranslateError

  • PyObject * PyExc_ValueError

  • PyObject * PyExc_ZeroDivisionError

  • PyObject * PyExc_BlockingIOError

  • PyObject * PyExc_BrokenPipeError

  • PyObject * PyExc_ChildProcessError

  • PyObject * PyExc_ConnectionError

  • PyObject * PyExc_ConnectionAbortedError

  • PyObject * PyExc_ConnectionRefusedError

  • PyObject * PyExc_ConnectionResetError

  • PyObject * PyExc_FileExistsError

  • PyObject * PyExc_FileNotFoundError

  • PyObject * PyExc_InterruptedError

  • PyObject * PyExc_IsADirectoryError

  • PyObject * PyExc_NotADirectoryError

  • PyObject * PyExc_PermissionError

  • PyObject * PyExc_ProcessLookupError

  • PyObject * PyExc_TimeoutError

  • PyObject * PyExc_EnvironmentError

  • PyObject * PyExc_IOError

  • PyObject * PyExc_WindowsError

  • PyObject * PyExc_RecursionErrorInst

  • PyObject * PyExc_Warning

  • PyObject * PyExc_UserWarning

  • PyObject * PyExc_DeprecationWarning

  • PyObject * PyExc_PendingDeprecationWarning

  • PyObject * PyExc_SyntaxWarning

  • PyObject * PyExc_RuntimeWarning

  • PyObject * PyExc_FutureWarning

  • PyObject * PyExc_ImportWarning

  • PyObject * PyExc_UnicodeWarning

  • PyObject * PyExc_BytesWarning

  • PyObject * PyExc_ResourceWarning

(https://hg.python.org/cpython/file/default/Include/setobject.h)

  • PyObject * _PySet_Dummy

(https://hg.python.org/cpython/file/default/Include/sliceobject.h)

  • PyObject _Py_EllipsisObject

(https://hg.python.org/cpython/file/default/Python/Python-ast.c)

  • PyObject *Load_singleton

  • PyObject *Store_singleton

  • PyObject *Del_singleton

  • PyObject *AugLoad_singleton

  • PyObject *AugStore_singleton

  • PyObject *Param_singleton

  • PyObject *And_singleton

  • PyObject *Or_singleton

  • PyObject *Add_singleton

  • PyObject *Sub_singleton

  • PyObject *Mult_singleton

  • PyObject *MatMult_singleton

  • PyObject *Div_singleton

  • PyObject *Mod_singleton

  • PyObject *Pow_singleton

  • PyObject *LShift_singleton

  • PyObject *RShift_singleton

  • PyObject *BitOr_singleton

  • PyObject *BitXor_singleton

  • PyObject *BitAnd_singleton

  • PyObject *FloorDiv_singleton

  • PyObject *Invert_singleton

  • PyObject *Not_singleton

  • PyObject *UAdd_singleton

  • PyObject *USub_singleton

  • PyObject *Eq_singleton

  • PyObject *NotEq_singleton

  • PyObject *Lt_singleton

  • PyObject *LtE_singleton

  • PyObject *Gt_singleton

  • PyObject *GtE_singleton

  • PyObject *Is_singleton

  • PyObject *IsNot_singleton

  • PyObject *In_singleton

  • PyObject *NotIn_singleton

caches/interning

...

free lists

...

most static const vars

  • won't change (and their data, if pointers, shouldn't change either)

static Python types

  • these should be okay because customizations should only be in .dict and not in C

static declaration arrays

  • these should be okay because they aren't touched by anything
  • <all PyStructSequence_* arrays>
  • <all PyTypeObject method/settergetter/etc. arrays>

_PyIdentifier and friends

  • shouldn't ever change

other static declarations

(https://hg.python.org/cpython/file/default/Include/pymacro.h)

  • #define PyDoc_VAR(...)
  • #define PyDoc_STRVAR(...)
  • #define PyDoc_STR(...)

(https://hg.python.org/cpython/file/default/Include/pyport.h)

  • #define Py_LOCAL(type)
  • #define Py_LOCAL_INLINE(type)

PEP 432

API Examples

feedback

personal correspondence (Nick)

On the other hand (to Elixir), a "SubinterpreterExecutor" for concurrent.futures,
and finally creating a "concurrent.pool" module with thread,
subinterpreter and multiprocessing backends would likely cover that, so
your mileage may vary.

Having a clearer path forward to multi-core exploitation without
serialisation and other IPC overhead that I can describe there will be a
potentially significant factor in changing perception of python-dev's
perception of the GIL problem.

Threads aren't the problem, shared memory threading without appropriate
access control primitives are the problem. Race conditions, especially
around object life cycles, are the bane of both correctness *and*
stability.

Rust gets this right by strictly controlling object ownership: you can
*hand over* a reference to another thread of control quite easily, but
*sharing* a reference is harder (and strongly discouraged).

zero-copy ownership based in-memory message passing system for
communication between C++ state machines

> You could also argue that in practice nearly all concurrent
> programming is IO-bound and that asyncio solves that for us.  Recent
> discussions (e.g. PEP 492) lead me to believe that it's not nearly
> that simple.

It's not, especially given the impact a single CPU bound thread can
have on IO latency."""

the model we should be aiming for is "the data storage separation of
multiprocessing, with the low message passing overhead of threading".

That gives us a lot more opportunity to create a hierarchical locking
model where each interpreter has a Local Interpreter Lock with the
characteristics of the GIL we know and love today, while the GIL itself
becomes a read/write lock where some operations that span multiple
interpreters still get serialised globally, but in the general case
subinterpreters can run in separate threads completely independently of
each other. This also gives us a path towards true multi-core threading
*without* getting rid of reference counting first.

There's also a potential connection to Petr's extension module loading
PEP here, as we could exploit that to allow extension modules to opt out
of being loaded in subinterpreters.

The reason subinterpreters get my vote is because improving them along
with improving embedding support in general has a host of beneficial
effects independently of whether or not we succeed in eliminating the
GIL.

In particular, the improved embedding support acts as a potential
carrot for the *animation* industry (particularly AutoDesk) to embark on
the migration to Python 3, rather than relying solely on the stick of
dropping Python 2 support.

We know from PyPy's adoption challenges that C extension compatibility
is non-negotiable, and from mod_wsgi that C extensions already kinda
sorta work (in most cases) in the context of subinterpreters.

personal correspondence (Brett)

it's memory copying + serialization vs lock issues. Keep it in the same
process and a good chunk of the message-passing problem goes away.

https://mail.python.org/pipermail/python-ideas/2015-June/034177.html

  • Yury Selivanov
    • +1 to an efficient way to share immutable objects
    • offer to help
  • Nathaniel Smith
    • shared-nothing threads + explicit sharing mechanism
    • concerns about backward-compat and memory safety
  • me
    • initially: share only strictly immutable objects
    • move refcounts to own table (memory page)
    • freeze object graph
  • Devin Jeanpierre
    • fork as an alternative
    • atfork registration
    • separate multiprocessing pickling from non-pickling code
    • freeze refcounts prior to a fork
    • multiprocessing.Value/Array as example
    • anything to facilitate sharing objects between threads applies to procs
    • +1 to an efficient way to share immutable objects
  • Chris Angelico
    • disallow forking is too restrictive?
  • me
    • only retrict os.fork()/multiprocessing
    • we will start out restricted and ease restrictions from there
  • Nick
    • subinterpreter support similar to JS web worker threads
    • mod_wsgi already demonstrates that subinterpreters mostly just work
    • extension modules can be problematic w/ subinterpreters
    • keeping subinterpreters subordinate to main interpreter (a la main thread)
    • ...simplifies designt and impl. aspects
    • main interpreter can do things subinterpreters can't
    • perhaps limit extension modules to main interpreter
  • Steven D'Aprano
    • restriction on Task arg to only callables not Pythonic?
    • reference to Armin Ronacher's blog post
    • too much shared between interpreters?
  • me
    • we should be able to kill the GIL per subinterpreter (if careful)
  • Nathaniel
    • need at least hand-wavy example
    • shared-nothing semantics limit options with mutable objects
    • ...(deep copy or real transfer where sender no longer has object a la rust)
    • requires self-contained object graph
    • you already cannot legally modify Python immutable objects (e.g. tuple)
    • C code is always able to break memory safety under shared-memory threading
    • ...(just make it easy not to)
    • refcounts and GC are a different matter
  • Nick
    • half-heartedly pushing for this for years
    • helping Graham investigate subinterpreter issues for years
    • mod_wsgi teaches us that core mechanism of subinterpreters is good
    • not that far off from ditching the GIL for subinterpreters
    • worst-case is memcpy-based message passing (better than pickling)
    • better: object-ownership-based system
    • enough prior art that SMOE assertion is justified
    • subinterpreters never exposed in Python because enough rough edges
    • without Python API subinterpreters are harder to test (so buggier)
    • without Python API subinterpreters are more CPython-specific
    • 3 tasks
      1. file off rough edges (incl. multi-Py-impl)
      2. primitives for safe & efficient message passing
      3. allow subinterpreters to truly execute in parallel on multi-core
    • each of those is useful on its own
    • allows incremental progress toward ultimate goal
  • Wes Turner
  • Guido
    • sounds like a good start
    • strict read-only sharing easiest (limiting risks)
    • refcounts still an issue
    • what about class definitions and import modules?
    • this is an important topic! (thanks for getting it started)
  • Antoine Pitrou
    • memcopy for sharing not good enough (object graphs)
  • Antoine
    • singleton objects and builtin types are shared (plus others in dark corners)
    • memory allocator is shared
  • me
    • look into an encapsulation for object graphs?
  • Stefan Behnel
    • support API by which objects can opt in to sharing (e.g. numpy arrays)
  • Antoine
    • what is the performance goal (relative to multiprocesing)?
  • Stefan
    • PEP 489 is related
    • Cython should be able to support subinterpreters fine
  • Nick
    • aim for zero-copy speed for at least known immutable values (& PEP 3118).
    • no slower than multiprocessing for anything.
  • Sturla Molden
    • allow independent interpreters within same process
    • one interpreter per thread (a la TCL and .NET)
    • for multi-core subinterpreters don't help much?
    • on POSIX fork is very fast + more efficient than serialization
    • multiprocessing only slow due to Windows (no fork)
    • you still eat the cost of starting up a subinterpreter + a thread
    • multi-processing counts as multi-core
    • why not just use IPC to share objects?
    • IPC is fast
    • MPI performs and scales better than OpenMP
    • refcounts are false shares unless using multiple processes
    • pickle makes multiprocessing slow, not IPC
    • IPC: pipes, sockets/named pipes/shared memory
    • shared memory just as fast as between subinterpreters?
    • look for way to share objects between processes
    • can't use simplified GIL API with subinterpeters (ctypes, cython)
  • Antoine
    • API compatibility does not allow independent interpreters API
    • complex and experimental topic: share graph of garbage-collected objects
    • ...between independent processes
    • PyGilState API is not subinterpreter-compatible
    • ...(issue10915, issue15751)
  • Devin
    • costly to initialize each subinterpreter (e.g. imports)
    • fork (with shared memory) is faster and uses less memory
    • subinterpreters are cross-platform
    • serialization is why multiprocessing is slow, not IPC
    • make is fast to share an object graph between independent executors
    • shared memory is instantaneous? (POSIX)
  • Devin
    • subinterpreters is practically the same thing as multiple processes
    • be sure to satisfy the concerns of folks that actually run parallel code
  • Nick
  • Stefan
    • C extension opt-in to not-protected-by-GIL on a per-function/method-basis
  • Nick
    • avoid serialization through shared references to immutable objects
    • ...(+ PEP 3118)
    • folks are free to work on improving multiprocessing's IPC efficiency
  • Sturla
    • shared memory can facilitate shared references with some low-level magic
  • Ron Adam
    • instead of freezing, modify a flag or counter if something's mutated
    • ...(turned off by default)
    • opt in to immutability via a flag (within a block)
  • Andrew Barnert
    • be less restrictive regarding forking within subinterpreters
    • are builtins and sys.modules shared between subinterpreters?
  • Andrew
    • allocate object graphs into the same memory page
  • Nick
    • semantics of object sharing should be as if copied but avoid actually copying
    • solve the problem for subinterpreters first because it is simpler
  • Nick
    • prototype some of the public API ideas using Jython and Vert.x
    • 2 layers: Python API + subinterpreter implementation
  • Greg Smith
    • if fork from subinterpreter, child proc must not return control to Python
  • Greg
    • subinterpreters have to re-import everything except extension modules
    • refcount + COW interplay is costly
    • subinterpreters currently similar to worker threads (amortize startup cost)
  • Nick
    • PEP 432 will help with configuring and isolating subinterpreters
  • Nathaniel
    • do some clever sharing to mitigate subinterpreter startup costs
  • Barry
    • Emacs' unexec (during build preload a bunch of always-used immutable modules
    • ...and freeze the state such that startup is much faster
  • Greg
    • Barry's idea would have to work with hash randomization
  • Devin
    • a patch to freeze refcounts before forking
    • a patch to move refcounts to own pages

https://news.ycombinator.com/item?id=9861966

resources

Python concurrency

the GIL

subinterpreters

mod_wsgi

subprocess

memory management

interpreter isolation

Python implementations

other languages

concurrency models

concurrency techniques

practical concurrency

"message" passing

CSP

distributed computing

immutability

other

Clone this wiki locally