Introduce the Hybrid ABI #371

antocuni · 2022-10-16T15:41:05Z

This PR introduces the hybrid ABI:

extensions which targets universal cannot #include <Python.h>, cannot use legacy features and produce a file called e.g. simple.hpy0.so.
extensions which targets hybrid can use Python.h and produce a file called e.g. simply.hpy0-cpy38.so.

See the updated docs for more info.

Filename
For universal, I choose to use .hpy0.so because we should use a versioned scheme. I didn't want to use .hpy1.so for now, because we might still want to change the ABI before becoming "official"

For hybrid, we want to include both the hpy version and the cpython version. I chose hpy0-cp38.so, which is the result of many sub-choices.
To start with, let's see what is the filename that you get on "normal" extensions:

linux: foo.cpython-38-x86_64-linux-gnu.so.
linux stable ABI: foo.abi3.so
mac: foo.cpython-310-darwin.so (or .dylib? I think I've seen both)
win: foo.cp38-win_amd64.pyd
pypy: foo.pypy38-pp73-x86_64-linux-gnu.so
graalpy: foo.graalpy-38-native-x86_64-darwin.dylib

It varies wildly:

cpython uses both cpython-38 and cp38.
the platform tag sometimes includes the CPU, sometimes not
the stable ABI extensions doesn't include the platform tag at all

So for hpy hybrid I chose to:

use cp38 instead of cpython-38: this is shorter, and most importantly it's the same tag which goes in the wheel names
don't include the platform tag: if it's not needed for the stable ABI, it means it's not needed for us as well

Compile time errors
In universal mode, you are not allowed to use Python.h: we try hard to display a nice compile time error by putting our own Python.h in the include dirs. This might not work in all setups, but it should be good enough for most cases.

In universal mode, you cannot call HPy_AsPyObject and HPy_FromPyObject: I managed to get a compile-time error on gcc, but not on old clangs nor in MSVC. Again, too bad but it's better than nothing:

hpy/hpy/devel/include/hpy/cpy_types.h

Lines 24 to 27 in 4cde717

    
           // declare the following API functions as _HPY_LEGACY, which triggers an 
        
           // #error if they are used 
        
           HPyAPI_FUNC _HPY_LEGACY cpy_PyObject *HPy_AsPyObject(HPyContext *ctx, HPy h); 
        
           HPyAPI_FUNC _HPY_LEGACY HPy HPy_FromPyObject(HPyContext *ctx, cpy_PyObject *obj);

hpy/hpy/devel/include/hpy.h

Lines 76 to 95 in 4cde717

    
           // clang and gcc supports __has_attribute, MSVC doesn't. This should be enough 
        
           // to be able to use it portably 
        
           #ifdef __has_attribute 
        
           #   define _HPY_compiler_has_attribute(x) __has_attribute(x) 
        
           #else 
        
           #   define _HPY_compiler_has_attribute(x) 0 
        
           #endif 
        
           #ifdef HPY_ABI_UNIVERSAL 
        
           #  if _HPY_compiler_has_attribute(error) 
        
                // gcc, clang>=14 
        
           #    define _HPY_LEGACY __attribute__((error("Cannot use legacy functions when targeting the HPy Universal ABI"))) 
        
           #  else 
        
                // we don't have any diagnostic feature, too bad 
        
           #    define _HPY_LEGACY 
        
           #  endif 
        
           #else 
        
              // in non-universal modes, we don't attach any attribute 
        
           #  define _HPY_LEGACY 
        
           #endif

The other legacy features that you cannot use are .legacy_methods and .legacy_slots: but these are now impossible to use, because in order to pass anything meaningful you have to instantiate e.g. a PyMethodDef struct, which you can't because you cannot include Python.h

We still need to reference some cpy_* types, e.g. to declare the HPyModule_Spec.legacy_methods field. This is done here:

hpy/hpy/devel/include/hpy/cpy_types.h

Lines 4 to 36 in 4cde717

    
           /* ~~~~~~~~~~~~~~~~ CPython legacy types ~~~~~~~~~~~~~~~~ */ 
        
           /* These are the types which are needed to implement legacy features such as 
        
              .legacy_slots, .legacy_methods, HPy_FromPyObject, HPy_AsPyObject, etc. 
        
              In cpython and hybrid ABI mode we can #include Python.h and use the "real" 
        
              types. 
        
              In universal ABI mode, legacy features cannot be used, but we still need 
        
              the corresponding C types to use in the HPy declarations. Note that we use 
        
              only forward declarations, meaning that it will actually be impossible to 
        
              instantiate any of these struct. 
        
           */ 
        
           #ifdef HPY_ABI_UNIVERSAL 
        
           typedef struct FORBIDDEN_cpy_PyObject cpy_PyObject; 
        
           typedef struct FORBIDDEN_PyMethodDef cpy_PyMethodDef; 
        
           typedef struct FORBIDDEN_bufferinfo cpy_Py_buffer; 
        
           // declare the following API functions as _HPY_LEGACY, which triggers an 
        
           // #error if they are used 
        
           HPyAPI_FUNC _HPY_LEGACY cpy_PyObject *HPy_AsPyObject(HPyContext *ctx, HPy h); 
        
           HPyAPI_FUNC _HPY_LEGACY HPy HPy_FromPyObject(HPyContext *ctx, cpy_PyObject *obj); 
        
           #else 
        
           // Python.h has already been included by the main hpy.h 
        
           typedef PyObject cpy_PyObject; 
        
           typedef PyMethodDef cpy_PyMethodDef; 
        
           typedef Py_buffer cpy_Py_buffer; 
        
           #endif /* HPY_ABI_UNIVERSAL */

in cpython and hybrid modes, we use the real types from Python.h. In universal mode, we use "fake" types, but note that they are only forward declarations and they cannot be instantiated.

This should also fix #288.

Future improvements
There are many things which we can to to improve the situation, but I think they can be done in future PRs:

post-compilation sanity check: after compiling to universal, we could run a sanity check to actually ensure that we don't link against any Py* symbol
runtime check: it would be nice to have two different context for universal and hybrid ABI: the universal ctx would fail cleanly if you try to call HPy_{As,From}PyObject or use .legacy_methods, etc.
better compile time errors on different compilers

…Instead of having cpython-ABI by default, we require it to be set explicitly. Also start to introduce supprt for the upcoming hybrid ABI

…distutils. Currently it's exactly the same as the universal ABI

… hybrid ABI mode

…ersal ABI

… override hpy_abi

…at we can use hybrid on some files

…lusive again, and adjust the codebase

Before this commit, we always used .hpy.so, even on windows. Now do a much better job: 1. we use .hpy0.so instead of .hpy.so. We will migrate to .hpy1.so as soon as the ABI is stable 2. we use the proper extension on non-linux systems 3. for the hybrid ABI, we use a different scheme, which generates e.g. .hpy0-cp38.so on CPython (but with lots of different cases for all possible systems, see abitag.py and its test)

… using it

…in universal mode

…ile time error

…al mode

fangerer

Great work, thank you!
I added some comments but I don't see blockers.

hpy/devel/abitag.py

hpy/devel/include/hpy.h

hpy/devel/include/hpy/cpy_types.h

hpy/devel/include/hpy/hpymodule.h

fangerer · 2022-11-03T08:25:41Z

hpy/universal/src/hpymodule.c

@@ -155,7 +155,7 @@ static PyObject *get_version(PyObject *self, PyObject *ignored)
 }

 static PyMethodDef HPyMethods[] = {
-    {"load", (PyCFunction)load, METH_VARARGS | METH_KEYWORDS, "Load a .hpy.so"},
+    {"load", (PyCFunction)load, METH_VARARGS | METH_KEYWORDS, "Load a .hpy0.so"},


We are also using this in hpy/devel/abitag.py. I think these two usages should have a strong connection. Can we do what we do with the version? Define it in a C header and expose it in Python.

One little more thing: there is an int ctx_version field in HPyContext. Matti brought to my attention, that this field basically is the ABI version, right? So, I think we should set ctx_version appropriately.

yes, good point. It turns out that sharing the same definition in C and Python is a mess as usual, so I opted for defining it in two places (hpy.h and abitag.py) and add a test which checks that they are on sync.
Also, I renamed ctx->ctx_version into ctx->abi_version and set its value to HPY_ABI_TAG, see commits 71d2a7d and 1f70e65.

test/conftest.py

fangerer · 2022-11-03T08:30:08Z

test/test_legacy_forbidden.py

+# We need this because some of the nice compilation errors (such as the ones
+# causes by _HPY_LEGACY) are triggered only by gcc. Would be nice to have them
+# also for other compilers
+ONLY_GCC = pytest.mark.skipif(sys.platform!='linux', reason='GCC only')


Should that maybe go into support.py ?

fixed by 4907ee8

steve-s · 2022-11-03T10:30:08Z

If we include the ABI version in the tag, I am wondering if we need #356. At the same time I like the approach in #356, because it will be "fool-proof" -- even if you have wrong tag for whatever reason it'll still check and choose the right ABI for your extension. So right now I am in favor of keeping both the ABI tag and #356 even though they are redundant.

fangerer · 2022-11-03T10:59:18Z

@steve-s You are right, I forgot about that effort. Would it make sense to introduce some generic ABI tag?

steve-s · 2022-11-03T13:48:36Z

Would it make sense to introduce some generic ABI tag?

What exactly do you mean by that? That we'd drop hpyX tag as suggested in this PR and use just generic hpy? I don't know. I am not opposed to having both the tag with a major ABI version (in the sense of #356) and the mechanism introduced in #356.

fangerer · 2022-11-03T13:51:48Z

What exactly do you mean by that? That we'd drop hpyX tag as suggested in this PR and use just generic hpy?

Yes, that's what I meant.

I do also think that a generic ABI tag does not make a lot of sense. I agree that we can have both.

hpy/devel/abitag.py

mattip · 2022-11-10T09:49:16Z

Is there more to do here or should we merge this as-is and incrementally improve things toward a release?

mattip · 2022-11-10T09:51:01Z

Closes #366

steve-s · 2022-11-10T13:26:31Z

I also had a cursory look over the changes and I vote for merging it and incrementally improve things toward a release

mattip · 2022-11-16T10:36:17Z

@antocuni would you like to make one last pass over the comments and clear the merge conflict?

…e the hpy_abi fixture

Co-authored-by: Florian Angerer <[email protected]>

mattip · 2022-11-23T18:09:24Z

after the capsule changes, this PR has conflicts

…BI_TAG match

antocuni · 2022-11-24T09:49:28Z

If we include the ABI version in the tag, I am wondering if we need #356. At the same time I like the approach in #356, because it will be "fool-proof" -- even if you have wrong tag for whatever reason it'll still check and choose the right ABI for your extension. So right now I am in favor of keeping both the ABI tag and #356 even though they are redundant.

if think we can have both: basically, the logic of this PR tries to ensure that you cannot use CPython symbols by mistake, and #356 would become a sanity check to ensure that it works well.
Not sure if it's high-priority, but it surely doesn't hurt.

…antocuni/hybrid-abi

… and fix conflicts about capsules

antocuni · 2022-11-24T09:57:21Z

@mattip @fangerer @steve-s I think I addressed all the comments and fixed the conflicts.
If you don't have anything against, I'll merge the PR as soon as the tests are green

fangerer · 2022-11-24T10:14:42Z

LGTM. Please go ahead and merge.
That's great work. Thank you!

mattip · 2022-11-24T10:42:42Z

Yay!

antocuni added 30 commits October 16, 2022 17:38

try to rationalize the macros which we use to select the target ABI. …

feb883b

…Instead of having cpython-ABI by default, we require it to be set explicitly. Also start to introduce supprt for the upcoming hybrid ABI

blindly try to fix cppcheck

e5077cb

try again to fix cppcheck

ffc2773

minimal support for the hybrid ABI, enough to pass the tests in test_…

6b0556d

…distutils. Currently it's exactly the same as the universal ABI

make --reuse-venv a real pytest option, it's easier to use

e5e188b

add a test that checks that we can use legacy features in cpython and…

7f93614

… hybrid ABI mode

try to actively forbid to #include <Python.h> when targeting the univ…

6953c78

…ersal ABI

Merge remote-tracking branch 'origin/master' into antocuni/hybrid-abi

d0850d8

put the LeakDetector into its own fixture, so that we can more easily…

79cb459

… override hpy_abi

introduce a way to change the desired ABIs on a per-file basis, so th…

3e0f29d

…at we can use hybrid on some files

use the hybrid ABI for these two files

2e95e10

there is no need to #include Python.h in this test

e6b9463

introduce the hybrid+debug ABI for tests

0892c8f

fix this

8ad1201

this doesn't work on windows, we need to investigate

daffce6

try a bit more to debug why it fails on windows

93a565a

MSVC prints the #error messages on stout, adjust the test for it

3518c2a

kill HPY_UNIVERSAL_STRICT_ABI, make UNIVERSAL and HYBRID mutually exc…

8380bd1

…lusive again, and adjust the codebase

rename HPY_CPYTHON_ABI & co into HPY_ABI_CPYTHON & co.

a801cfc

remove the Makefile: it was clearly outdated and wrong, and nobody is…

460dc66

… using it

fix this macro

0d0d75a

this file must be compatible with py27, fix it

aaae66f

fix this occurence of the macro

b1e5ff6

try to blindly fix the documentation tests

015be7d

use the real CPython types in cpython and hybrid mode, and fake ones …

0b10c9d

…in universal mode

improve the docs and document the new hybrid ABI

dc64d71

replace all the remaining occurrences of Python/C with Python.h

d9fc276

add a test to check that the compilation fails with the expected comp…

650c140

…ile time error

raise compile-time errors if you use HPy_{As,From}PyObject in univers…

83c761e

…al mode

antocuni changed the title ~~WIP: introduce the Hybrid ABI~~ Introduce the Hybrid ABI Oct 20, 2022

antocuni marked this pull request as ready for review October 20, 2022 14:46

fangerer approved these changes Nov 3, 2022

View reviewed changes

mattip reviewed Nov 10, 2022

View reviewed changes

hpy/devel/abitag.py Show resolved Hide resolved

antocuni and others added 3 commits November 22, 2022 11:02

Merge remote-tracking branch 'origin/master' into antocuni/hybrid-abi

c80c527

move TestHPyCapsuleLegacy to its own file, because we need to overrid…

e2894ec

…e the hpy_abi fixture

Update hpy/devel/include/hpy/hpymodule.h

f20b181

Co-authored-by: Florian Angerer <[email protected]>

antocuni added 3 commits November 24, 2022 10:14

add a test to check that the C and Py constants for ABI_VERSION and A…

71d2a7d

…BI_TAG match

rename ctx->ctx_version into ctx->abi_version and test it

1f70e65

move the ONLY_LINUX marker to support.py

4907ee8

antocuni added 2 commits November 24, 2022 10:49

Merge branch 'antocuni/hybrid-abi' of github.com:hpyproject/hpy into …

b3c9295

…antocuni/hybrid-abi

Merge remote-tracking branch 'origin/master' into antocuni/hybrid-abi…

5e1d47b

… and fix conflicts about capsules

antocuni merged commit 3c85aa2 into master Nov 24, 2022

mattip mentioned this pull request Nov 24, 2022

Consider let build fail when hpy-abi=universal and Py* symbols are discovered #366

Closed

fangerer mentioned this pull request Nov 24, 2022

Make HPy compatible with Python 3.11 #381

Merged

This was referenced Dec 1, 2022

Add HPy ABI version in the ABI and check it when loading a module #356

Closed

Emit HPy ABI version into the binary and check it when loading a module #387

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce the Hybrid ABI #371

Introduce the Hybrid ABI #371

antocuni commented Oct 16, 2022 •

edited

Loading

fangerer left a comment

fangerer Nov 3, 2022

fangerer Nov 11, 2022

antocuni Nov 24, 2022

fangerer Nov 3, 2022

antocuni Nov 24, 2022

steve-s commented Nov 3, 2022

fangerer commented Nov 3, 2022

steve-s commented Nov 3, 2022

fangerer commented Nov 3, 2022

mattip commented Nov 10, 2022

mattip commented Nov 10, 2022

steve-s commented Nov 10, 2022

mattip commented Nov 16, 2022

mattip commented Nov 23, 2022

antocuni commented Nov 24, 2022

antocuni commented Nov 24, 2022

fangerer commented Nov 24, 2022

mattip commented Nov 24, 2022

	// declare the following API functions as _HPY_LEGACY, which triggers an
	// #error if they are used
	HPyAPI_FUNC _HPY_LEGACY cpy_PyObject HPy_AsPyObject(HPyContext ctx, HPy h);
	HPyAPI_FUNC _HPY_LEGACY HPy HPy_FromPyObject(HPyContext ctx, cpy_PyObject obj);

	// clang and gcc supports __has_attribute, MSVC doesn't. This should be enough
	// to be able to use it portably
	#ifdef __has_attribute
	# define _HPY_compiler_has_attribute(x) __has_attribute(x)
	#else
	# define _HPY_compiler_has_attribute(x) 0
	#endif

	#ifdef HPY_ABI_UNIVERSAL
	# if _HPY_compiler_has_attribute(error)
	// gcc, clang>=14
	# define _HPY_LEGACY __attribute__((error("Cannot use legacy functions when targeting the HPy Universal ABI")))
	# else
	// we don't have any diagnostic feature, too bad
	# define _HPY_LEGACY
	# endif
	#else
	// in non-universal modes, we don't attach any attribute
	# define _HPY_LEGACY
	#endif

	/* ~~~~~~~~~~~~~~~~ CPython legacy types ~~~~~~~~~~~~~~~~ */

	/* These are the types which are needed to implement legacy features such as
	.legacy_slots, .legacy_methods, HPy_FromPyObject, HPy_AsPyObject, etc.

	In cpython and hybrid ABI mode we can #include Python.h and use the "real"
	types.

	In universal ABI mode, legacy features cannot be used, but we still need
	the corresponding C types to use in the HPy declarations. Note that we use
	only forward declarations, meaning that it will actually be impossible to
	instantiate any of these struct.
	*/

	#ifdef HPY_ABI_UNIVERSAL

	typedef struct FORBIDDEN_cpy_PyObject cpy_PyObject;
	typedef struct FORBIDDEN_PyMethodDef cpy_PyMethodDef;
	typedef struct FORBIDDEN_bufferinfo cpy_Py_buffer;

	// declare the following API functions as _HPY_LEGACY, which triggers an
	// #error if they are used
	HPyAPI_FUNC _HPY_LEGACY cpy_PyObject HPy_AsPyObject(HPyContext ctx, HPy h);
	HPyAPI_FUNC _HPY_LEGACY HPy HPy_FromPyObject(HPyContext ctx, cpy_PyObject obj);

	#else

	// Python.h has already been included by the main hpy.h
	typedef PyObject cpy_PyObject;
	typedef PyMethodDef cpy_PyMethodDef;
	typedef Py_buffer cpy_Py_buffer;

	#endif /* HPY_ABI_UNIVERSAL */

Introduce the Hybrid ABI #371

Introduce the Hybrid ABI #371

Conversation

antocuni commented Oct 16, 2022 • edited Loading

fangerer left a comment

Choose a reason for hiding this comment

fangerer Nov 3, 2022

Choose a reason for hiding this comment

fangerer Nov 11, 2022

Choose a reason for hiding this comment

antocuni Nov 24, 2022

Choose a reason for hiding this comment

fangerer Nov 3, 2022

Choose a reason for hiding this comment

antocuni Nov 24, 2022

Choose a reason for hiding this comment

steve-s commented Nov 3, 2022

fangerer commented Nov 3, 2022

steve-s commented Nov 3, 2022

fangerer commented Nov 3, 2022

mattip commented Nov 10, 2022

mattip commented Nov 10, 2022

steve-s commented Nov 10, 2022

mattip commented Nov 16, 2022

mattip commented Nov 23, 2022

antocuni commented Nov 24, 2022

antocuni commented Nov 24, 2022

fangerer commented Nov 24, 2022

mattip commented Nov 24, 2022

antocuni commented Oct 16, 2022 •

edited

Loading