dev call 20201203

Dev Call: 3 December 2020

Present

Antonio, Armin, Florian, Matti, Ronan, Simon, Victor

Calling Python functions from HPy

HPy needs a API function or functions to allow calling Python functions and methods. The old C API provides many such methods (PyObject_Call, etc) so we need to decide which ones to support. A proposal has was made -- see https://github.com/hpyproject/hpy/issues/122.

After much discussion the concensus was to only implement HPy_Call for now and add other functions later as use cases arose. This keeps the HPy API simple for now and its easier to add new functions than to remove old ones.

ACTION: Just implement HPy_Call for now.

Argument Clinic for HPy

The discussion of HPy_Call lead to a discussion of the signatures of methods in extensions implemented with HPy. The idea of providing a mechanism like Argument Clinic for use in HPy extensions is intriguing for a few reasons.

Argument Clinic provides a very convenient and consistent way to document Python methods in C extensions and to convert the Python arguments into C arguments to pass to the function implementation.

The abstraction provided by a system like Argument Clinic would also allow the generation of multiple entry points with different function signatures. For example, both the ordinary Python extension method and a function that takes the C arguments directly could be exposed, allowing the parsing of arguments and intermediate Python objects to be completely omitted in situations where the C values are already available (e.g. inside Cython code, or a JIT).

There was general support for this idea, but it should remain completely optional so that porting existing C extensions does not require more work than necessary.

ACTION: Discuss more when there is a concrete proposal.

NumPy, Cython and HPy

Some members of the NumPy and Cython community are really looking forward to HPy and there is an attempt to find funding to do a trial port of NumPy to HPy.

ACTION: Be very excited!

Recent updates to packaging HPy universal modules

There is a pull request open to make HPy universal modules importable -- https://github.com/hpyproject/hpy/pull/124. This writes a small stub .py file that loads the compiled HPy universal shared library.

The compatibility with PyPy and PyPy's test suite still needs to be checked and any issues addressed, but first PyPy needs to be brought up to date with the other recent HPy changes.

On PyPy we should avoid the use of importlib in untranslated tests (it is very slow untranslated) and skip the FatalError test if we can't find a sensible way to make it work in both the CPython and PyPy tests.

Reorganising the HPy API headers

Currently public_api.h is the source of the HPy API definition that is used to autogenerate various concrete. It has some deficiencies though:

• It's not a real header for the API -- those have to be generated from it.
• It's not a complete header.
• It contains additional definitions that are not part of the API (e.g. constants like h_None, fake typedefs for HPy).

Since different universal C extensions should be built from exactly the same headers (for a given version of HPy), it would be good to have a fixed header that we can use. This would also allow us to easily point to what the HPy API is (and to document it).

Broadly there is a proposal to split public_api.h into multiple files, move it out of hpy.tools and into hpy.devel and add documentation to it. A thin layer would then have to be written that would allow implementations of HPy to walk the header definition and generate code needed by their implementation. This might encounter problems (e.g. pycparser might not be able to extract what we need from the real headers).

ACTION: We should try the proposed approach and see what happens.

Discussion of the HPy manifesto

We did a general review of the HPy manifesto:

In places the manifesto refers to "we" without making clear who "we" are. Probably "we" refers to those involved in HPy, but we should make that clearer or find a way to avoid such uses.

Our target audience is C extension module developers (and developers of tools for extension developers -- e.g. Cython). Right now such developers are the people who can try out HPy and give us feedback, and so are the most important audience to convince to try HPy right now.

The section on "Why do this?" needs further explanation of how changing the C API will allow each of the listed items to be potentially addressed. For example, sub-interpreters require avoiding global state, Gilectomy discovered that reference counting is a blocker to acceptable performance once the GIL is removed, the specialized implementation of lists requires preventing direct access to list internals.

ACTION: Add an explanation of why a lower-level API is not necessarily better for performance.

ACTION: We need to publicize the manifesto. Once we are happy with it we should add it to the HPy docs, convert it into an informational PEP, and post it to python-dev.

Microbenchmarks

Microbenchmarks were discussed during the manifest discussion as a means to alay performance concerns. We have a small microbenchmarks suite at the moment, but we should add a few more common idioms to it and advertise the results a bit more.

Currently the results of the microbenchmarks are very good. On CPython, HPy is as fast as the existing C API (i.e. there is no performance overhead). On PyPy the story is even better -- HPy is as fast as the existing C API on CPython (i.e. much faster than cpyext).

Note: We have the microbenchmarks primarily as a means of avoiding small mistakes that lead to performance issues. They are not intended to prove that HPy is fast -- that will come later when we try port more existing extensions to HPy.

ACTION: Add a microbenchmark for accessing items within a tuple.

ACTION: Add a microbenchmark for dictionary lookup.

ACTION: Mention the microbenchmarks in the manifesto.