New sysconfig API: Build information - Compilation

[FFY00]

Feature or enhancement

sysconfig should export all the information required to build extensions.

Pitch

The new API should try to expose the information in a compiler-agnostic way.

Information to expose

• Python version
  • currently: sys.version_info
• Interpreter version
  • currently: sys.implementation
• Extension suffixes
  • definition: See PEP 3149
    • consideration: PEP 3149 only defines the names for POSIX
      • Windows and other platforms still seem to follow it mostly (refer to packaging.tags's implementation)
      • Windows
    • design: The API should identify the different extensions types (interpreter, stable abi, none)
  • currently: importlib.machinery.EXTENSION_SUFFIXES, sysconfig.get_config_var('EXT_SUFFIX'), sysconfig.get_config_var('SHLIB_SUFFIX')
• Is a shared libpython available? What's its name? Location?
  • definition: Yes if built with --enable-shared
  • currently:
    • (available?) bool(sysconfig.get_config_var('LDLIBRARY'))
    • (name?) sysconfig.get_config_var('LDLIBRARY')
    • (location?) sysconfig.get_config_var('LIBDIR')
• Is a static libpython available? What's its name? Location?
  • definition: Yes unless built with --without-static-libpython
  • currently:
    • (available?) bool(sysconfig.get_config_var('LIBRARY'))
    • (name?) sysconfig.get_config_var('LIBRARY')
    • (location?) sysconfig.get_config_var('LIBDIR')
• Should extensions link against libpython?
  • definition: Yes on Android and Cygwin, no everywhere else (see extension built with a shared python cannot be loaded with a static python #65735)
  • currently: bool(sysconfig.get_config_var('LIBPYTHON'))
• Is it a debug build?
  • definition: Yes if built with --with-pydebug
  • currently: bool(sysconfig.get_config_var('Py_DEBUG'))
• C flags needed when using the C API
  • definition:
    • -fwrapv (until Enable -fstrict-overflow #96821)
      • Currently, we have unsafe code, and for a lack of a better option we assume it made into C macros, so extensions that use the C API should use -fwrapv.
• what else? please comment...

"definition" sections list the answer, if a question, or the value(s) of the item. "currently" sections list the current way of fetching the information.

This is list is a work in progress. Currently, we need to gather all the information that is required to build extensions. If you see anything that's missing, please comment!

Previous discussion

https://discuss.python.org/t/building-extensions-modules-in-a-post-distutils-world/23938
https://discuss.python.org/t/what-information-is-useful-to-know-statically-about-an-interpreter/25563
https://gregoryszorc.com/docs/python-build-standalone/20230116/distributions.html

New sysconfig API meta-issue: GH-103480

[FFY00]

It seems maturin needs the pointer width (see https://github.com/PyO3/maturin/blob/5d5b96a9974eac26b8cdf601cd2faf64f4999de9/src/python_interpreter/sysconfig-linux.json#L10).

[zooba]

Explicitly capturing the platform (not just the pointer width) that the runtime is targeting would be great. This is probably an arbitrary string used by the build toolset, rather than something universal.

Because Windows OS can emulate earlier architectures, and ARM64 and AMD64 have the same pointer width, the only way to know right now that you've got (in particular) an ARM64 build is to look at sys.winver. The running interpreter on an ARM64 processor could be x86 or AMD64, and the processor may be reported as one of these for emulation purposes, depending on how you ask.

Capturing the target platform at compile time would be much better.

[katietz]

I would see such feature as very useful to be able to use host's python interpreter in crosstool scenario. I would be very useful to have a switch/configuration-file, which can be used to setup python-interpreter settings ( platform.* sys.platform and co) so it matches the cross-target. Actually I ran into this issue while looking into emscripten/pyodide, which suffers exactly by this right now

[mattip]

For extension suffixes, there should be a way to expose the limited API name as well. xref this discussion on meson build, which resorts to using

from importlib.machinery import EXTENSION_SUFFIXES
limited_api_suffix = EXTENSION_SUFFIXES[1]

[FFY00]

For extension suffixes, there should be a way to expose the limited API name as well.

Yes. All the entries in importlib.machinery.all_extensions() would be provided by name in the API.