Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allocating extra data after an object instance #103743

Closed
jbradaric opened this issue Apr 24, 2023 · 0 comments
Closed

Allocating extra data after an object instance #103743

jbradaric opened this issue Apr 24, 2023 · 0 comments
Labels
topic-C-API type-feature A feature request or enhancement

Comments

@jbradaric
Copy link
Contributor

jbradaric commented Apr 24, 2023

Feature or enhancement

Add an unstable (in PEP 689 sense) function that allocates memory for a GC object and allows the caller to specify the size of additional memory that should be allocated after the object. Similar to what could be done using _PyObject_GC_Malloc function in Python <= 3.10.

Pitch

Currently there is no way of allocating extra data after an object and still being able to have a standard Python class that supports GC, allows inheriting and doesn't come with limitations of PyVarObject (see #103740).

See the linked discussion for more details, but in essence the use case is the following. We have a base class and metaclass that support defining "slots" on classes (similar to Python __slots__, but with a different behavior, more on that later). The data for the slots is stored after the object, in the same block of memory. When inheriting from a class that defines such "slots", the subclass gets all the slots from its base classes + slots defined on it. The data for all those slots is always stored after the object, so the slot storage size doesn't really come into calculation for tp_basicsize.

A nice effect of this behavior is that a subclass can inherit from multiple classes with each of the bases defining its own slots. The subclass then gets all those slots, as if they were directly defined on it (in contrast with Python __slots__, which prevents such inheritance).

CC: @encukou

Previous discussion

See this discussion for more details.

Linked PRs

@jbradaric jbradaric added the type-feature A feature request or enhancement label Apr 24, 2023
encukou added a commit that referenced this issue May 2, 2023
Co-authored-by: Petr Viktorin <[email protected]>
Co-authored-by: Erlend E. Aasland <[email protected]>
@encukou encukou closed this as completed May 2, 2023
carljm added a commit to carljm/cpython that referenced this issue May 2, 2023
* main:
  pythongh-103822: [Calendar] change return value to enum for day and month APIs (pythonGH-103827)
  pythongh-65022: Fix description of tuple return value in copyreg (python#103892)
  pythonGH-103525: Improve exception message from `pathlib.PurePath()` (pythonGH-103526)
  pythongh-84436: Add integration C API tests for immortal objects (pythongh-103962)
  pythongh-103743: Add PyUnstable_Object_GC_NewWithExtraData (pythonGH-103744)
  pythongh-102997: Update Windows installer to SQLite 3.41.2. (python#102999)
  pythonGH-103484: Fix redirected permanently URLs (python#104001)
  Improve assert_type phrasing (python#104081)
  pythongh-102997: Update macOS installer to SQLite 3.41.2. (pythonGH-102998)
  pythonGH-103472: close response in HTTPConnection._tunnel (python#103473)
  pythongh-88496: IDLE - fix another test on macOS (python#104075)
  pythongh-94673: Hide Objects in PyTypeObject Behind Accessors (pythongh-104074)
  pythongh-94673: Properly Initialize and Finalize Static Builtin Types for Each Interpreter (pythongh-104072)
  pythongh-104016: Skip test for deeply neste f-strings on wasi (python#104071)
carljm added a commit to carljm/cpython that referenced this issue May 2, 2023
* main: (760 commits)
  pythonGH-104102: Optimize `pathlib.Path.glob()` handling of `../` pattern segments (pythonGH-104103)
  pythonGH-104104: Optimize `pathlib.Path.glob()` by avoiding repeated calls to `os.path.normcase()` (pythonGH-104105)
  pythongh-103822: [Calendar] change return value to enum for day and month APIs (pythonGH-103827)
  pythongh-65022: Fix description of tuple return value in copyreg (python#103892)
  pythonGH-103525: Improve exception message from `pathlib.PurePath()` (pythonGH-103526)
  pythongh-84436: Add integration C API tests for immortal objects (pythongh-103962)
  pythongh-103743: Add PyUnstable_Object_GC_NewWithExtraData (pythonGH-103744)
  pythongh-102997: Update Windows installer to SQLite 3.41.2. (python#102999)
  pythonGH-103484: Fix redirected permanently URLs (python#104001)
  Improve assert_type phrasing (python#104081)
  pythongh-102997: Update macOS installer to SQLite 3.41.2. (pythonGH-102998)
  pythonGH-103472: close response in HTTPConnection._tunnel (python#103473)
  pythongh-88496: IDLE - fix another test on macOS (python#104075)
  pythongh-94673: Hide Objects in PyTypeObject Behind Accessors (pythongh-104074)
  pythongh-94673: Properly Initialize and Finalize Static Builtin Types for Each Interpreter (pythongh-104072)
  pythongh-104016: Skip test for deeply neste f-strings on wasi (python#104071)
  pythongh-104057: Fix direct invocation of test_super (python#104064)
  pythongh-87092: Expose assembler to unit tests (python#103988)
  pythongh-97696: asyncio eager tasks factory (python#102853)
  pythongh-84436: Immortalize in _PyStructSequence_InitBuiltinWithFlags() (pythongh-104054)
  ...
facebook-github-bot pushed a commit to facebookincubator/cinderx that referenced this issue Aug 5, 2024
…ExtraData

Summary: `_PyObject_GC_Malloc` has gone away in 3.12, but I think the reason we're using it - adding extra data to a GC-managed `PyObject*` - is covered by the recently added `PyUnstable_Object_GC_NewWithExtraData`. The future of this [is a little uncertain](python/cpython#103743) but for now it seems to do what we want.

Reviewed By: alexmalyshev

Differential Revision: D60684943

fbshipit-source-id: d6b8fb001502c68805bd7edd31fdec61eb9389bb
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
topic-C-API type-feature A feature request or enhancement
Projects
None yet
Development

No branches or pull requests

3 participants