C API: Add PyWeakref_GetRef() function

[vstinner]

PyWeakref_GET_OBJECT() and PyWeakref_GetObject() return a borrowed reference to the object, or a borrowed reference to None if the object has been finalized. This API is error-prone and not easy to use.

This API was discussed in 2016: https://mail.python.org/archives/list/[email protected]/thread/6BIPI4MCMAZFLJNIZB4JLTBW2COCACQQ/

I propose adding a new PyWeakref_GetRef() C API function which returns a new strong reference to the object, or NULL if the object has been finalized.

Linked PRs

• gh-105927: Refactor weakrefobject.c #105928
• gh-105927: Add _PyWeakref_GET_REF() internal function #105929
• gh-105927: Add PyWeakref_GetRef() function #105932
• gh-105927: _abc and _thread use PyWeakref_GetRef() #105961
• gh-105927: type_from_ref() uses _PyWeakref_GET_REF() #105963
• gh-105927: _ctypes use PyWeakref_GetRef() #105964
• gh-105927: _ssl uses _PyWeakref_GET_REF() #105965
• gh-105927: Fix test_weakref_capi() refleak #105966
• gh-105927: finalize_modules_clear_weaklist() uses _PyWeakref_GET_REF() #105971
• gh-105927: Add _PyWeakref_IS_DEAD() function #105992
• gh-105927: Avoid calling PyWeakref_GET_OBJECT() #105997
• gh-105927: PyImport_AddModule() uses _PyWeakref_GET_REF() #106001
• gh-105927: _ssl GET_SOCKET() uses _PyWeakref_GET_REF() #106002
• gh-105927: Deprecate PyWeakref_GetObject() function #106006
• gh-105927: Remove _PyWeakref_GetWeakrefCount() #106007
• gh-105927: PyWeakref_GetRef() returns 1 on success #106561
• gh-105927: Limit PyWeakref_GetRef() to limited C API 3.13 #117091

[erlend-aasland]

[...] or NULL if the object has been finalized.

With an error set? If no, would you need to call PyErr_Occurred() to find out if an error has occurred? See also capi-workgroup/problems#1

[vstinner]

With an error set? If no, would you need to call PyErr_Occurred() to find out if an error has occurred? See also capi-workgroup/problems#1

No error is set. See PR #105929 for the first part of the implementation.

[erlend-aasland]

No error is set. See PR #105929 for the first part of the implementation.

Would it be better to return None if the object has been finalized? The idiom is that APIs returning a PyObject pointer only will return NULL in case of error.

[vstinner]

Would it be better to return None if the object has been finalized? The idiom is that APIs returning a PyObject pointer only will return NULL in case of error.

It's possible to return a strong reference to None if the reference is dead. But it makes the API harder to handle since you have to handle Py_None reference count in the "dead reference" code path.

In Python 3.12, Py_None is immortal. But I plan to add PyWeakref_GetRef() to https://pythoncapi-compat.readthedocs.io/ and on Python 3.11 and older, you have to carefully Py_DECREF(Py_None).

[encukou]

Consider making this int PyWeakref_GetRef(PyObject *weakref, PyObject **value), returning -1 on error (with exception set), 0 on dead reference, and 1 for success (with value set).
See capi-workgroup/problems#1 and especially #105201 (comment) for discussion.

[vstinner]

The proposed function cannot fail. If you use it on the wrong type, it does crash instead.

[encukou]

I see. Different reasoning then.
Please don't add infallible functions to the stable ABI. There is a possiblity that it will need to fail in the future.

[erlend-aasland]

I agree with Petr; if the internals of the API changes in the future, it would be unfortunate if we also had to add a PyWeakRef_GetRefWithError API as well. However, we take this possibility in account, when designing new APIs.

[erlend-aasland]

See also capi-workgroup/problems#20

[vstinner]

I'm trying to fix race conditions by avoiding borrow references. If the API is modified so it can raise an exeption, it means that all callers must be updated to handle exceptions? That sounds like a lot of work.

I don't see the point of making the API so it can raise an exeption, if there is no known case where it can happen.

I would prefer to not modify PyWeakref_GetRef() to emit a cute exception if the argument is not a weak reference. I prefer to move this responsibility to the caller. The Modules/_abc.c code is an interesting example: it handles the rare case if an object which is not a weakref landed in the container.

I would prefer to have an API simple to use when the caller controls the argument object type.

[encukou]

I'm trying to fix race conditions by avoiding borrow references. If the API is modified so it can raise an exeption, it means that all callers must be updated to handle exceptions? That sounds like a lot of work.

Yes.

I would prefer to have an API simple to use when the caller controls the argument object type.

Please don't add it to the stable ABI. Maintaining that for decades is also a lot of work, and avoiding the work now means we'll pay in the future.

I would prefer to not modify PyWeakref_GetRef() to emit a cute exception if the argument is not a weak reference.

That's not necessary.
I'm asking for it to be possible to add an error case in the future. For example, on an implementation that uses tagged pointers, you could get a MemoryError when getting the object.

[vstinner]

Ok, Erlend and Petr convinced me, I updated my PR to use the API:

int PyWeakref_GetRef(PyObject *ref, PyObject **pobj)

[vstinner]

Oh, PEP 703 – Making the Global Interpreter Lock Optional in CPython proposes PyWeakref_FetchObject() which is more or less the PyWeakref_GetRef() function that was just added to Python 3.13.

[vstinner]

I added PyWeakref_GetRef() functions and deprecated PyWeakref_GetObject() and PyWeakref_GET_OBJECT() functions. I close the issue.