Unknown Python implementation: pypy38

[oscarbenjamin]

This comes from a SciPy issue: scipy/scipy#16737

As of SciPy 1.9.0 pip install scipy fails under PyPy with the error message:

+ meson install --destdir /tmp/pip-install-pkqrdcy5/scipy_8a1b991ddd494020a114650533cb4ba6/.mesonpy-ve4kjypi/install
      Traceback (most recent call last):
        File "/opt/hostedtoolcache/PyPy/3.8.13/x64/lib/pypy3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 156, in prepare_metadata_for_build_wheel
          hook = backend.prepare_metadata_for_build_wheel
      AttributeError: module 'mesonpy' has no attribute 'prepare_metadata_for_build_wheel'
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "/tmp/pip-build-env-mupbdlja/overlay/lib/pypy3.8/site-packages/mesonpy/__init__.py", line 714, in _calculate_file_abi_tag_heuristic_posix
          return mesonpy._tags.StableABITag(tag)
        File "/tmp/pip-build-env-mupbdlja/overlay/lib/pypy3.8/site-packages/mesonpy/_tags.py", line 39, in __init__
          raise ValueError(f'Invalid PEP 3149 stable ABI tag, expecting pattern `{self._REGEX.pattern}`')
      ValueError: Invalid PEP 3149 stable ABI tag, expecting pattern `^abi(?P<abi_number>[0-9]+)$`
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "/opt/hostedtoolcache/PyPy/3.8.13/x64/lib/pypy3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
        File "/opt/hostedtoolcache/PyPy/3.8.13/x64/lib/pypy3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/opt/hostedtoolcache/PyPy/3.8.13/x64/lib/pypy3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 160, in prepare_metadata_for_build_wheel
          whl_basename = backend.build_wheel(metadata_directory, config_settings)
        File "/tmp/pip-build-env-mupbdlja/overlay/lib/pypy3.8/site-packages/mesonpy/__init__.py", line 927, in build_wheel
          return project.wheel(out).name
        File "/tmp/pip-build-env-mupbdlja/overlay/lib/pypy3.8/site-packages/mesonpy/__init__.py", line 875, in wheel
          wheel = _WheelBuilder(self).build(self._install_plan, self._copy_files, self._build_dir)
        File "/tmp/pip-build-env-mupbdlja/overlay/lib/pypy3.8/site-packages/mesonpy/__init__.py", line 356, in build
          wheel_file = pathlib.Path(directory, f'{self.name}.whl')
        File "/tmp/pip-build-env-mupbdlja/overlay/lib/pypy3.8/site-packages/mesonpy/__init__.py", line 165, in name
          python_tag=self._project.python_tag,
        File "/tmp/pip-build-env-mupbdlja/overlay/lib/pypy3.8/site-packages/mesonpy/__init__.py", line 657, in python_tag
          selected_tag = self._select_abi_tag()
        File "/tmp/pip-build-env-mupbdlja/overlay/lib/pypy3.8/site-packages/mesonpy/__init__.py", line 779, in _select_abi_tag
          tags = self._files_by_tag()
        File "/tmp/pip-build-env-mupbdlja/overlay/lib/pypy3.8/site-packages/mesonpy/__init__.py", line 759, in _files_by_tag
          tag = self._calculate_file_abi_tag_heuristic(file)
        File "/tmp/pip-build-env-mupbdlja/overlay/lib/pypy3.8/site-packages/mesonpy/__init__.py", line 723, in _calculate_file_abi_tag_heuristic
          return self._calculate_file_abi_tag_heuristic_posix(filename)
        File "/tmp/pip-build-env-mupbdlja/overlay/lib/pypy3.8/site-packages/mesonpy/__init__.py", line 716, in _calculate_file_abi_tag_heuristic_posix
          return mesonpy._tags.LinuxInterpreterTag(tag)
        File "/tmp/pip-build-env-mupbdlja/overlay/lib/pypy3.8/site-packages/mesonpy/_tags.py", line 79, in __init__
          f'Unknown Python implementation: {self.implementation}. '
      NotImplementedError: Unknown Python implementation: pypy38. Please report this to https://github.com/FFY00/mesonpy/issues and include information about the Python distribution you are using.

You can see the CI setup for this up to the line that fails here:
https://github.com/sympy/sympy/blob/7754d037d97d56de9e1bcecb81ecadd4a2e683be/.github/workflows/runtests.yml#L170-L208

The logs for the failed SciPy build are here:
https://github.com/sympy/sympy/runs/7583332309?check_suite_focus=true

[mattip]

I think the problem is that this line https://github.com/FFY00/meson-python/blob/dba737a6f77db75a70f2e211586dd59a3f915732/mesonpy/_tags.py#L108

should be elif self.implementation.startswith('pypy'):

but a cursory glance elsewhere in the file shows https://github.com/FFY00/meson-python/blob/dba737a6f77db75a70f2e211586dd59a3f915732/mesonpy/_tags.py#L121

so tests are needed to make sure these paths work properly. Maybe adding a pypy-3.8 run to the pythons tested https://github.com/FFY00/meson-python/blob/dba737a6f77db75a70f2e211586dd59a3f915732/.github/workflows/tests.yml#L23-L29

would trigger some failures?

[FFY00]

Hi @oscarbenjamin, sorry for the delay getting back to you. The PyPy tag handling seems to be incomplete.

@mattip how are ABI tags PyPy 3.8 formatted?

[mattip]

PyPy uses pypy38_pp73 and pypy39_pp73 for the python3.8 and python3.9 versions respectively. The pp73 part comes from the "PyPy" version, which is currently 7.3.9. All the 7.3.x PyPy versions are ABI compatible. This comes from normalize(sysconfig.get_config_var('SOABI') where normalize converts - to _.

[rgommers]

gh-135 should fix this. would be great if someone can test that they indeed see that the problem is gone.

[mattip]

Is there a smaller meson-python project than scipy that I can try out?

[rgommers]

You could try the test suite, install meson-python from the PR branch into your env and then either pytest . or

$ cd tests/package/purelib-and-platlib
$ python -m build . --no-isolation

[rgommers]

This was fixed, and there is CI for PyPy 3.8 and 3.9 now. The upcoming v0.9.0 release will contain the fix. So I'll close this.