dev call 20220407

Dev Call: 7 April 2022

Present

Antonio, Simon, Matti, Ronan, Florian, Tim, Stepan

Agenda

• HPy release 0.0.4
• Autogen: separate API and ABI (context)

• HPy binary compatibility

  • Assuming that our ABI will always be backwards compatible is a bit naïve. We should start thinking about how to deal with that.
  • Inspired by SDL2 paper: https://www.reddit.com/r/linux_gaming/comments/1upn39/sdl2_adds_dynamic_api_magic_to_allow_updating_it/ (sorry, couldn't find the original paper)
  • We should think about a policy for supported HPy ABI versions

• Debug mode: The possibility to decide per module makes implementation complicated. Maybe a global decision would be enough (i.e. all contexts are debug contexts or none).
• HPyType_FromSpec: disallow setting any tp_flags not supported by HPy https://github.com/hpyproject/hpy/issues/305
• Remove support for Python 3.6 https://github.com/hpyproject/hpy/issues/307

• NumPy Progress and Plans

  • NpyIter
  • UFunc
  • Wiki with some migration notes: https://github.com/hpyproject/hpy/wiki/Numpy-HPy-migration-notes:-blockers-and-concerns

• Open HPy PRs

  • Capsule PR: https://github.com/hpyproject/hpy/pull/308
  • exact type check: https://github.com/hpyproject/hpy/pull/309
  • detect HPy_Close/return of context constants in debug: https://github.com/hpyproject/hpy/pull/303

HPy Release 0.0.4

• HPy release 0.0.4: no objections; just do it.
• PyPy also started to integrate HPy 0.0.4 already.
• GraalPython is almost ready. Next official release will be in July but we provide snapshot builds for experimenting.

• We will ignore (for now) that HPyGlobals cannot be shared between multiple incompatible (sub-)interpreters

  • Implementations can choose to allow sharing of HPyGlobals between their own subintereperters. For example, in CPython we should make an indirection through interpreter state to support that.
  • Sharing between multiple Python implementations: the contract would be to (atomically) write the address of HPyGlobal as its value. The address is then used by the implementations as a token/identifier for given global. We can decide to go for this later.

• Reason (for the release): The GraalPython team migrated MPL and Kiwisolver to HPy and wants to make those public.

Autogen: separate API and ABI (context)

• Autogen API/ABI: we should have a real public header that contains the official API.
• People agree, that separation basically makes sense.
• Concerns are that API/ABI might run out of sync.
• Create and issue and continue discussion there: https://github.com/hpyproject/hpy/issues/311

HPy binary compatibility

• There is some discussion in the Python community. So maybe it's a good time to think about it and make first steps.
• Our binary stability is that we have the context and if we don't change the functions and just add at the end, we are fine. But that's probably not future-proof.
• The HPy context already has a version.
• Rough idea from Tim/Florian how to do it: * The extension somehow needs to request the required context version. * In multi-phase init, the module can request the context version in the spec. * In single-phase init, we need some _protocol_ to request the HPy context in the right version.
• Idea by Antonio: The extension could just provide some symbol (much like HPyInit_*) that would specify the context. We can generate the symbol in macro HPy_MODINIT.
• What if an extension calls function of different extension and passes the context. They could both require a different context version. We will need some _trampoline_ or some converter function to call C functions (with HPyContext) of a different extension.
• The CPython approach is: if you can load and call a function that raises an exception, then the extension can at least handle the incompatibility nicely (without an immediate crash).
• Tim and Florian prefer the approach of being able to request a specific context version because that would allow to emulate obsolete/removed ABI functions (maybe at the cost of performance).
• Simon: how hard would it be for impls support older ABI?

Misc

• HPyGlobal: we are seeing a lot of load global - call API fun - close handle and these could maybe be done more efficiently by having a dedicated API for it. For example, in our NumPy
• Simon: talked to two students that will do some HPy benchmarking later this year. We invite them to IRC, mailing list, and dev calls with next month.
• Cython: Florian didn't address the last review comments yet. The PR is still open but priorities are now on Numpy migration. He will eventually finish that ASAP.

Preliminary Benchmark Results for Kiwi (enaml_bench) and Matplotlib (mpl-bench basic benchmark)

Matplotlib CPython C-API, CPython HPy C-ABI, CPython HPy universal

Matplotlib CPython C-API, CPython HPy C-ABI, CPython HPy universal, GraalPython HPy Universal on LLVM, GraalPython C-API on LLVM

Kiwi CPython HPy C-ABI, CPython C-API, CPython HPy universal

Kiwi CPython HPy C-ABI, CPython HPy universal, CPython HPy C-API, GraalPython HPy Universal Native, GraalPython HPy Universal on LLVM