gh-91744: Add semi-stable C API tier #91789

encukou · 2022-04-21T15:56:45Z

This adds the semi-stable API tier witth a few initial functions/types. Adding more is left to another PR.

d'oh!

vstinner

Is PyInterpreterState_SetEvalFrameFunc() really usable in the semi-stable API? The second parameter type comes from the internal C API. There is nothing to inspect a _PyInterpreterFrame structure in the semi-stable API.

I don't know how debuggers use PyInterpreterState_SetEvalFrameFunc().

cc @markshannon

vstinner · 2022-04-21T16:54:40Z

See also #91906

h-vetinari

Some comments from taking a look out of curiosity

Include/README.rst

Doc/whatsnew/3.11.rst

h-vetinari · 2022-04-28T03:23:06Z

Doc/c-api/stable.rst

+There are two tiers of API with different stability exepectations,
+enabled by specific macros:
+
+- :c:macro:`Py_USING_SEMI_STABLE_API` exposes API that may change
+  without deprecation warnings.
+- :c:macro:`Py_LIMITED_API` exposes API that is compatible across
+  several minor releases.


exepectations -> expectations

One could also say: "there are ~~two~~ public tiers of the API" (emphasis just to show addition); underscored resp. Py_BUILD_CORE are arguably a tier as well...?

I've struck out the "two" because I think it would be clearer to expose the full hierarchy, à la:

Suggested change

There are two tiers of API with different stability exepectations,

enabled by specific macros:

- :c:macro:`Py_USING_SEMI_STABLE_API` exposes API that may change

without deprecation warnings.

- :c:macro:`Py_LIMITED_API` exposes API that is compatible across

several minor releases.

There are three public tiers of the API with different stability

expectations, two of which need to be enabled by specific macros:

- :c:macro:`Py_LIMITED_API` exposes API that is compatible across

several minor releases.

- (no macro): default C-API covered by :pep:`387`.

- :c:macro:`Py_USING_SEMI_STABLE_API` exposes API that may change

without deprecation warnings between minor releases.

I'd also suggest to lead with the most stable one.

By “There are two tiers of API with different stability expectations” I wanted to present them as being additional, different from the default that's covered in the preceding paragraphs.
The order matches the detailed sections below, which would be somewhat tricky to reorder -- the one-paragraph semi-stable tier would feel lost if it was at the end.

Doc/c-api/stable.rst

Doc/c-api/code.rst

h-vetinari · 2022-04-28T03:25:00Z

Doc/c-api/code.rst

+   use :c:func:`PyCode_NewEmpty` instead.
+
+   Since the definition of the bytecode changes often, calling
+   :c:func:`PyCode_New` directly can bind you to a precise Python version.


style: better "can directly" than "directly can" (or directly at the very end)

It's not “can directly bind you”, but “calling :c:func:PyCode_New directly” (as opposed to PyCode_Replace for example)

h-vetinari · 2022-04-28T03:27:34Z

Misc/NEWS.d/next/C API/2022-04-21-17-25-22.gh-issue-91744.FgvaMi.rst

@@ -0,0 +1,10 @@
+Defining :c:macro:`Py_USING_SEMI_STABLE_API` will expose "semi-stable API",


Perhaps it's worth considering to stay consistent with the spelling of "semi-stable" / "semistable". Currently the include folders are named like the latter (which I think looks good in a path), mentions in the docs like here often have the hyphen.

Even though the hyphen is grammatically slightly more correct, my personal preference would be without.

Which also opens the question if the macro should be Py_USING_SEMI_STABLE_API or Py_USING_SEMISTABLE_API

I still have hope that someone will come up with a better name. Python-dev is usually so good at bikeshedding!
I'll postpone renaming for now.

encukou · 2022-04-28T14:05:36Z

Thank you for the comments!

vstinner

I expected the API and macro to mention "unstable" rather than "semi-stable" (Py_USING_SEMI_STABLE_API => Py_USING_UNSTABLE_API).

vstinner · 2022-05-04T10:05:34Z

Include/cpython/code.h

-        PyObject *, PyObject *, PyObject *, int, PyObject *,
-        PyObject *);
-        /* same as struct above */
+/* See Include/semi-stable/code.h for PyCode_New* */


I'm not sure that this comment is useful.

Suggested change

/* See Include/semi-stable/code.h for PyCode_New* */

/* See Include/unstable/code.h for PyCode_New* */

vstinner · 2022-05-04T10:06:14Z

Include/pyport.h

@@ -61,6 +61,10 @@
 #  define Py_BUILD_CORE
 #endif

+// Expose unstable API when building Python
+#ifdef Py_BUILD_CORE
+#define Py_USING_UNSTABLE_API


Suggested change

#define Py_USING_UNSTABLE_API

# define Py_USING_UNSTABLE_API

vstinner · 2022-05-04T10:06:34Z

Include/pyport.h

+#ifdef Py_USING_UNSTABLE_API
+#define _Py_NEWLY_UNSTABLE(VERSION_UNUSED)
+#else
+#define _Py_NEWLY_UNSTABLE(VERSION) Py_DEPRECATED(VERSION)
+#endif


Suggested change

#ifdef Py_USING_UNSTABLE_API

#define _Py_NEWLY_UNSTABLE(VERSION_UNUSED)

#else

#define _Py_NEWLY_UNSTABLE(VERSION) Py_DEPRECATED(VERSION)

#endif

#ifdef Py_USING_UNSTABLE_API

# define _Py_NEWLY_UNSTABLE(VERSION_UNUSED)

#else

# define _Py_NEWLY_UNSTABLE(VERSION) Py_DEPRECATED(VERSION)

#endif

vstinner · 2022-05-04T10:08:15Z

Include/unstable/code.h

+        PyObject *, PyObject *, PyObject *, PyObject *,
+        PyObject *, PyObject *, PyObject *, int, PyObject *,
+        PyObject *);
+        /* same as struct PyCodeObject */


I suggest to move the comment before the function definition: "same parameters as PyCodeObject structure members".

vstinner · 2022-05-04T10:09:20Z

Include/unstable/pystate.h

+
+/* Frame evaluation API (added for PEP-523) */
+
+struct _PyInterpreterFrame;


Is this needed?

vstinner · 2022-05-04T10:13:35Z

Doc/c-api/stable.rst

+
+- :c:macro:`Py_USING_SEMI_STABLE_API` exposes API that may change
+  without deprecation warnings.
+- :c:macro:`Py_LIMITED_API` limits exposed API to API that is compatible


Is API repeated on purpose?

Suggested change

- :c:macro:`Py_LIMITED_API` limits exposed API to API that is compatible

- :c:macro:`Py_LIMITED_API` limits exposed API that is compatible

vstinner · 2022-05-04T10:15:21Z

Include/cpython/code.h

@@ -219,6 +201,11 @@ typedef enum _PyCodeLocationInfoKind {
    PY_CODE_LOCATION_INFO_NONE = 15
 } _PyCodeLocationInfoKind;

+#define Py_UNSTABLE_CODE_H
+#include "unstable/code.h"
+#undef Py_UNSTABLE_CODE_H


I would prefer to include it in top-level Include/code.h, as done in Include/pystate.h.

vstinner · 2022-05-04T10:18:13Z

Doc/whatsnew/3.11.rst

+  - :c:func:`PyInterpreterState_GetEvalFrameFunc`
+  - :c:func:`PyInterpreterState_SetEvalFrameFunc`
+  - :c:type:`PyFrameEvalFunction`
+


Maybe mention also _PyFrameEvalFunction type which is renamed?

encukou · 2022-08-10T09:55:15Z

I'm deferring the PEP. Thanks for the comments, I'll take them into account next time I try.

encukou added 6 commits April 21, 2022 17:31

Add the semi-stable C-API tier, with PEP-523 functions & PyCode_New*

0185c42

Document Py_USING_SEMI_STABLE_API & functions that need it

381fc11

Add PyFrameEvalFunction & related functions to semi-stable API

23e8a1d

Docs: Expand the intro for C API stability

02c5466

Add blurb

e8e1884

Add What's New entry

a441699

encukou requested review from vstinner and ncoghlan April 21, 2022 15:56

encukou requested a review from markshannon as a code owner April 21, 2022 15:56

bedevere-bot added the awaiting core review label Apr 21, 2022

encukou added 2 commits April 21, 2022 17:59

Merge main branch to resolve a conflict

31beafd

And one more fix for the conflict

132fd8e

encukou marked this pull request as draft April 21, 2022 16:07

encukou mentioned this pull request Apr 21, 2022

PEP 689 -- Add an unstable C-API tier #91744

Closed

19 tasks

Exclude semistable/ from the limited API

7dba38c

d'oh!

vstinner reviewed Apr 21, 2022

View reviewed changes

vstinner mentioned this pull request Apr 26, 2022

gh-91719: Add pycore_opcode.h internal header file #91906

Merged

h-vetinari reviewed Apr 28, 2022

View reviewed changes

Address some review suggestions

74419bd

encukou added 2 commits April 28, 2022 16:13

Merge main branch to resolve conflicts

31ea0db

Fix issue reference in docs

75abb20

vstinner mentioned this pull request May 2, 2022

[C API] Move the PyCodeObject structure to the internal C API (make the structure opaque) #91397

Closed

encukou added 2 commits May 4, 2022 01:16

Rename semi-stable to unstable

b189c81

Merge main branch to resolve conflicts

911f78a

vstinner reviewed May 4, 2022

View reviewed changes

encukou closed this Aug 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-91744: Add semi-stable C API tier #91789

gh-91744: Add semi-stable C API tier #91789

encukou commented Apr 21, 2022

vstinner left a comment

vstinner commented Apr 21, 2022

vstinner commented Apr 26, 2022

h-vetinari left a comment

h-vetinari Apr 28, 2022

encukou Apr 28, 2022

h-vetinari Apr 28, 2022

encukou Apr 28, 2022 •

edited

Loading

h-vetinari Apr 28, 2022

h-vetinari Apr 28, 2022

encukou Apr 28, 2022

encukou commented Apr 28, 2022

vstinner left a comment

vstinner May 4, 2022

vstinner May 4, 2022

vstinner May 4, 2022

vstinner May 4, 2022

vstinner May 4, 2022

vstinner May 4, 2022

vstinner May 4, 2022

vstinner May 4, 2022

encukou commented Aug 10, 2022

		@@ -0,0 +1,10 @@
		Defining :c:macro:`Py_USING_SEMI_STABLE_API` will expose "semi-stable API",

	/* See Include/semi-stable/code.h for PyCode_New* */
	/* See Include/unstable/code.h for PyCode_New* */


		/* Frame evaluation API (added for PEP-523) */

		struct _PyInterpreterFrame;

	- :c:macro:`Py_LIMITED_API` limits exposed API to API that is compatible
	- :c:macro:`Py_LIMITED_API` limits exposed API that is compatible

gh-91744: Add semi-stable C API tier #91789

gh-91744: Add semi-stable C API tier #91789

Conversation

encukou commented Apr 21, 2022

vstinner left a comment

Choose a reason for hiding this comment

vstinner commented Apr 21, 2022

vstinner commented Apr 26, 2022

h-vetinari left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

encukou Apr 28, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

encukou commented Apr 28, 2022

vstinner left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

encukou commented Aug 10, 2022

encukou Apr 28, 2022 •

edited

Loading