Unknown Python implementation: pyston (cannot install SciPy >= 1.9.0 on Pyston) #142

PLPeeters · 2022-09-14T17:00:30Z

This is similar to #121, only it's for the lesser-known (but awesome) Pyston.

NotImplementedError: Unknown Python implementation: pyston. Please report this to https://github.com/FFY00/mesonpy/issues and include information about the Python distribution you are using.

Could you add support for it?

The text was updated successfully, but these errors were encountered:

FFY00 · 2022-09-14T18:11:52Z

We can look into it, do you have some wheels that you can share? I need to see the values that are expected for Pyston.

PLPeeters · 2022-09-15T05:50:29Z

Sure, here's a ZIP file with two wheels built for Pyston: wheels.zip

rgommers · 2022-09-27T12:29:03Z

Thanks @PLPeeters. To make this easier for readers, here are the details. Wheel file names:

adns-1.4_py0-pyston38-pyston_23_x86_64_linux_gnu-linux_x86_64.whl
cffi-1.15.1-pyston38-pyston_23_x86_64_linux_gnu-linux_x86_64.whl

Content of the first wheel:

├── adns-1.4_py0.dist-info
│   ├── METADATA
│   ├── RECORD
│   ├── top_level.txt
│   └── WHEEL
├── ADNS.py
├── adns.pyston-23-x86_64-linux-gnu.so
└── DNSBL.py

Content of the second wheel:

.
├── cffi
│   ├── api.py
│   ├── backend_ctypes.py
│   ├── _cffi_errors.h
│   ├── _cffi_include.h
│   ├── cffi_opcode.py
│   ├── commontypes.py
│   ├── cparser.py
│   ├── _embedding.h
│   ├── error.py
│   ├── ffiplatform.py
│   ├── __init__.py
│   ├── lock.py
│   ├── model.py
│   ├── parse_c_type.h
│   ├── pkgconfig.py
│   ├── recompiler.py
│   ├── setuptools_ext.py
│   ├── vengine_cpy.py
│   ├── vengine_gen.py
│   └── verifier.py
├── cffi-1.15.1.dist-info
│   ├── entry_points.txt
│   ├── LICENSE
│   ├── METADATA
│   ├── RECORD
│   ├── top_level.txt
│   └── WHEEL
└── _cffi_backend.pyston-23-x86_64-linux-gnu.so

I'm not sure that these wheels are actually correct. The pyston-23 part looks tricky. PyPy doesn't have a hyphen there, it uses tags like pypy38, pypy39.

FFY00 · 2022-09-28T18:34:33Z

I'm not sure that these wheels are actually correct.

To me, it looks reasonable. These values were probably first defined by distutils/setuptools, and it makes sense that for unknown implementations, it uses the extension name as the ABI tag -- we could probably do the same in the future, but for now we have enough issues 😅

Hey @kmod, could you clarify if these wheel tags are correct? Is there any other guidance you may want to provide regarding Pyston wheels? Thanks!

I'm not sure that these wheels are actually correct. The pyston-23 part looks tricky. PyPy doesn't have a hyphen there, it uses tags like pypy38, pypy39.

The 2.3 is the Pyston version, PyPy also includes it in the tags, just in a slightly different format.

My takeaways from these wheels are:

The Python tag is pyston{sysconfig.get_config_var('py_version_nodot')}
The ABI tag is ext_name.lower()

kmod · 2022-09-28T20:53:56Z

If I remember correctly the abi tag situation was changed recently (less than a year ago). We requested a 2-letter ABI tag, since this was previously required, but the python packaging team decided to move away from the 2-letter tag scheme. I forget the exact details of what they changed, and I was unable to find the thread where this happened.

@undingen do you remember more about this?

undingen · 2022-09-29T20:22:01Z

I'm not following what this thread is about but maybe this information about the Pyston PEP425 tag is helpful:

This was the Thread where we tried to get out own tag but ended up not getting/doing it: https://discuss.python.org/t/pep425-python-tag-for-pyston/13039
So if I remember correctly we did not end up changing the tag - just sticked to the default tag.

This was quite a long time ago so I can't really remember so I took a quick look how the wheel filename gets calculated:
The packaging package does all the work in this path packaging.tags.generic_tags():
https://github.com/pypa/packaging/blob/5f46d1532f45ae32cd7831c6f45212f7066c1ed0/packaging/tags.py#L254

The PEP425 tag comes down to:
<interpreter>-<abi>-<platform>
Where:

interpreter is pyston38: sys.implementation.name + sysconfig.get_config_var("py_version_nodot")
abi is pyston_23_x86_64_linux_gnu: sysconfig.get_config_var("SOABI").replace(".", "_").replace("-", "_")
platform is linux_x86_64: sysconfig.get_platform().replace(".", "_").replace("-", "_")

Or the other way around:

>>> from packaging.tags import parse_tag
>>> t = list(parse_tag("pyston38-pyston_23_x86_64_linux_gnu-linux_x86_64"))[0]
>>> t.interpreter
'pyston38'
>>> t.abi
'pyston_23_x86_64_linux_gnu'
>>> t.platform
'linux_x86_64'

Hope this helps

jelber2 · 2022-10-19T10:48:20Z

Ran into the same error as the OP. Any progress?

dnicolodi · 2022-11-02T09:07:46Z

I think #191 should solve this issue. Can someone running Pyston test?

belm0 · 2022-11-03T08:37:30Z

@dnicolodi happy to try it, as I've just hit this issue today. But how do I go about having pip wheel scipy use a development branch of meson-python?

dnicolodi · 2022-11-03T10:10:08Z

IIRC pip wheel does not use an isolated environment for the build, thus you can install meson-python from the PR git branch in an virtual environment and execute pip wheel from there.

rgommers · 2022-11-03T10:44:12Z

I usually explicitly turn off build isolation:

# install the meson-python version you want to test, and other build dependencies
python -m build --no-isolation --skip-dependency-check

belm0 · 2022-11-03T11:12:48Z

not sure what to make of this

I'm using pip wheel --no-build-isolation scipy==1.9.3 after installing meson-python from the branch.

      Installing /tmp/pip-wheel-06r0aanq/scipy_f0e7592b9c36444d945d8242174601c2/scipy/io/_harwell_boeing/tests/test_hb.py to /tmp/pip-wheel-06r0aanq/scipy_f0e7592b9c36444d945d8242174601c2/.mesonpy-5txqoc6k/install/usr/lib/python3.8-pyston2.3/site-packages/scipy/io/_harwell_boeing/tests
      Traceback (most recent call last):
        File "/home/venv/lib/python3.8-pyston2.3/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 144, in prepare_metadata_for_build_wheel
          hook = backend.prepare_metadata_for_build_wheel
      AttributeError: module 'mesonpy' has no attribute 'prepare_metadata_for_build_wheel'

      During handling of the above exception, another exception occurred:
...
        File "/home/venv/lib/python3.8-pyston2.3/site-packages/mesonpy/__init__.py", line 317, in _use_stable_abi
          abis.append(m.group('abi'))
      AttributeError: 'str' object has no attribute 'group'

dnicolodi · 2022-11-03T11:16:45Z

Obviously I introduced a bug with one of the last tweaks. Let me fix it. I'll be back to you ASAP

belm0 · 2022-11-03T11:19:45Z

Thank you. Pyston is trivial to install from a .deb if you need it.

https://github.com/pyston/pyston/releases/tag/pyston_2.3.5

dnicolodi · 2022-11-03T11:20:21Z

Silly bug. Fixed. Can you please try again?

@rgommers can you please approve the CI tests run for #191?

belm0 · 2022-11-03T12:24:27Z

appears to work, thank you!

  Created wheel for scipy: filename=scipy-1.9.3-pyston38-pyston_23_x86_64_linux_gnu-manylinux_2_31_x86_64.whl size=24178195 sha256=83bbafea0e7ac2862728c488fb745f72991706f87688b53932be3abf37f60f21

dnicolodi · 2022-11-03T12:31:20Z

Wow, hat's a mouthful of a wheel filename. Does it install correctly?

belm0 · 2022-11-03T12:34:03Z

it installs correctly, though I won't test my app until tomorrow

on scipy 1.8.3 the pyston wheel was huge-- 80 MB for some reason. 1.9.3 is down to 24MB (smaller than CPython wheel)

belm0 · 2022-11-03T23:34:05Z

confirmed that the 1.9.3 pyston package works in our app

Use the wheel contents only to determine whether the wheel contains python ABI dependent modules or other platform dependent code. Fixes mesonbuild#142, mesonbuild#189, mesonbuild#190.

Use the wheel contents only to determine whether the wheel contains python ABI dependent modules or other platform dependent code. Fixes mesonbuild#142, fixes mesonbuild#189, fixes mesonbuild#190.

rgommers · 2022-11-04T08:20:09Z

@rgommers can you please approve the CI tests run for #191?

done. thanks for working on this!

on scipy 1.8.3 the pyston wheel was huge-- 80 MB for some reason. 1.9.3 is down to 24MB (smaller than CPython wheel)

That will be due to stripping of the extension modules, which is done with meson-python correctly, but required manual stripping with distutils.

Use the wheel contents only to determine whether the wheel contains python ABI dependent modules or other platform dependent code. Fixes mesonbuild#142, fixes mesonbuild#189, fixes mesonbuild#190.

The extension modules filename suffixes do not contain enough information to correctly determine the wheel tags. Instead introspect the Python interpreter to derive the wheel tags. This is the same approach used by other PEP517 backends, most notably wheel. The wheel contents only to determine whether the wheel contains python ABI dependent modules or other platform dependent code. The packaging module is the reference wheel tags derivation implementation and it is used (or vendored) by most python packages dealing with wheels. However, the API provided by packaging is cumbersome to use for our purposes and, with the goal of merging this code into Meson in the future, it is good to avoid an additional dependency. Therefore, the tags derivation code is reimplemented. Tests are added to verify that the tags produced by meson-python agree with the ones produced by packaging to ensure that the two implementations will not diverge. Fixes mesonbuild#142, fixes mesonbuild#189, fixes mesonbuild#190.

jelber2 · 2023-02-14T10:38:11Z

I ran into this issue with of unknown Python implementation with pyston and had to do

pyston -m pip wheel --no-build-isolation scikit-learn
pyston -m pip install scikit_learn-1.2.1-pyston38-pyston_23_x86_64_linux_gnu-linux_x86_64.whl

instead of pyston -m pip install scikit-learn.

Is it because of the --no-build-isolation that it worked?

dnicolodi · 2023-02-14T11:09:57Z

AFAIK, scikit-learn does not use meson-python thus your issue is not related to this bug report. scikit-learn uses setuptools, thus this is most likely a bug in setuptools. scikit-learn pins setuptools version to < 60.0 for some reason. Most likely your issue is solved in more recent setuptools versions. Please check with the scikit-learn maintainers and eventually report a bug to setuptools.

rgommers · 2023-02-14T11:24:46Z

Scikit-learn depends on SciPy though, it has a build-requires of "scipy>=1.3.2",. That said, this fix is in meson-python 0.12.0 though, which should be used when SciPy 1.10.0 is built.

jelber2 · 2023-02-14T12:28:50Z

Ok, thanks, please close again I guess

dnicolodi · 2023-02-14T15:08:54Z

It would be nice to see the build log to understand where things go wrong. From the report it did not seem that what is failing is setting up the build environment (ie installing scipy in there).

jelber2 · 2023-02-14T15:35:06Z

Ok, tomorrow, I hope to check

jelber2 · 2023-02-15T11:08:29Z

micromamba create -n strainy-pyston3 -c bioconda -c conda-forge -c pyston samtools=1.14 bcftools=1.14 graphviz pyston
micromamba activate strainy-pyston3

# EDIT - update to proper .yaml format
micromamba env export

name: strainy-pyston3
channels:
- https://conda.anaconda.org/bioconda
- https://conda.anaconda.org/conda-forge
- https://conda.anaconda.org/pyston
dependencies:
- _libgcc_mutex=0.1=conda_forge
- _openmp_mutex=4.5=2_gnu
- atk-1.0=2.38.0=hd4edc92_1
- bcftools=1.14=hde04aa1_1
- bzip2=1.0.8=h7f98852_4
- c-ares=1.18.1=h7f98852_0
- ca-certificates=2022.12.7=ha878542_0
- cairo=1.16.0=ha61ee94_1014
- expat=2.5.0=h27087fc_0
- font-ttf-dejavu-sans-mono=2.37=hab24e00_0
- font-ttf-inconsolata=3.000=h77eed37_0
- font-ttf-source-code-pro=2.038=h77eed37_0
- font-ttf-ubuntu=0.83=hab24e00_0
- fontconfig=2.14.2=h14ed4e7_0
- fonts-conda-ecosystem=1=0
- fonts-conda-forge=1=0
- freetype=2.12.1=hca18f0e_1
- fribidi=1.0.10=h36c2ea0_0
- gdk-pixbuf=2.42.8=hff1cb4f_1
- gettext=0.21.1=h27087fc_0
- giflib=5.2.1=h36c2ea0_2
- graphite2=1.3.13=h58526e2_1001
- graphviz=7.0.5=h2e5815a_0
- gsl=2.7=he838d99_0
- gtk2=2.24.33=h90689f9_2
- gts=0.7.6=h64030ff_2
- harfbuzz=6.0.0=h8e241bc_0
- htslib=1.14=h9753748_2
- icu=70.1=h27087fc_0
- jpeg=9e=h0b41bf4_3
- keyutils=1.6.1=h166bdaf_0
- krb5=1.20.1=hf9c8cef_0
- lerc=3.0=h9c3ff4c_0
- libblas=3.9.0=16_linux64_openblas
- libcblas=3.9.0=16_linux64_openblas
- libcurl=7.87.0=h6312ad2_0
- libdeflate=1.10=h7f98852_0
- libedit=3.1.20191231=he28a2e2_2
- libev=4.33=h516909a_1
- libffi=3.4.2=h7f98852_5
- libgcc-ng=12.2.0=h65d4601_19
- libgd=2.3.3=h18fbbfe_3
- libgfortran-ng=12.2.0=h69a702a_19
- libgfortran5=12.2.0=h337968e_19
- libglib=2.74.1=h606061b_1
- libgomp=12.2.0=h65d4601_19
- libiconv=1.17=h166bdaf_0
- libnghttp2=1.51.0=hdcd2b5c_0
- libnsl=2.0.0=h7f98852_0
- libopenblas=0.3.21=pthreads_h78a6416_3
- libpng=1.6.39=h753d276_0
- librsvg=2.54.4=h7abd40a_0
- libssh2=1.10.0=haa6b8db_3
- libstdcxx-ng=12.2.0=h46fd767_19
- libtiff=4.4.0=h0fcbabc_0
- libtool=2.4.7=h27087fc_0
- libuuid=2.32.1=h7f98852_1000
- libwebp=1.2.4=h522a892_0
- libwebp-base=1.2.4=h166bdaf_0
- libxcb=1.13=h7f98852_1004
- libxml2=2.10.3=h7463322_0
- libzlib=1.2.13=h166bdaf_4
- ncurses=6.2=h58526e2_4
- openssl=1.1.1t=h0b41bf4_0
- pango=1.50.12=hd33c08f_1
- pcre2=10.40=hc3806b6_0
- perl=5.32.1=2_h7f98852_perl5
- pip=23.0=pyhd8ed1ab_0
- pixman=0.40.0=h36c2ea0_0
- pthread-stubs=0.4=h36c2ea0_1001
- pyston=2.3.4=5
- pyston2.3=2.3.4=14_23_pyston
- python=3.8.12=5_23_pyston
- python_abi=3.8=3_23_pyston
- readline=8.1=h46c0cb4_0
- samtools=1.14=hb421002_0
- setuptools=67.3.1=pyhd8ed1ab_0
- sqlite=3.37.0=h9cd32fc_0
- tk=8.6.12=h27826a3_0
- tzdata=2022g=h191b570_0
- wheel=0.38.4=pyhd8ed1ab_0
- xorg-kbproto=1.0.7=h7f98852_1002
- xorg-libice=1.0.10=h7f98852_0
- xorg-libsm=1.2.3=hd9c2040_1000
- xorg-libx11=1.7.2=h7f98852_0
- xorg-libxau=1.0.9=h7f98852_0
- xorg-libxdmcp=1.1.3=h7f98852_0
- xorg-libxext=1.3.4=h7f98852_1
- xorg-libxrender=0.9.10=h7f98852_1003
- xorg-renderproto=0.11.1=h7f98852_1002
- xorg-xextproto=7.3.0=h7f98852_1002
- xorg-xproto=7.0.31=h7f98852_1007
- xz=5.2.6=h166bdaf_0
- zlib=1.2.13=h166bdaf_4
- zstd=1.5.2=h3eb15da_6


pyston -m pip install pygraphviz matplotlib biopython gfapy edlib pysam
pyston -m pip install scikit-learn

Failing installation command

pyston -m pip install scikit-learn --force

            + meson setup --native-file=/tmp/pip-install-h4586j_l/scipy_d49dfd58e21a4dc0aa66b134666fecba/.mesonpy-native-file.ini -Ddebug=false -Doptimization=2 --prefix=/users/jean.elbers/micromamba/envs/strainy-pyston3 /tmp/pip-install-h4586j_l/scipy_d49dfd58e21a4dc0aa66b134666fecba /tmp/pip-install-h4586j_l/scipy_d49dfd58e21a4dc0aa66b134666fecba/.mesonpy-dnrdx37d/build
            + meson compile
            + meson install --destdir /tmp/pip-install-h4586j_l/scipy_d49dfd58e21a4dc0aa66b134666fecba/.mesonpy-dnrdx37d/install
            WARNING Using heuristics to map files to wheel, this may result in incorrect locations
            WARNING Could not tell if file was meant for purelib or platlib, so it was mapped to platlib: /tmp/pip-install-h4586j_l/scipy_d49dfd58e21a4dc0aa66b134666fecba/.mesonpy-dnrdx37d/install/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/__config__.py (/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/__config__.py)
            WARNING Using heuristics to map files to wheel, this may result in incorrect locations
            WARNING Could not tell if file was meant for purelib or platlib, so it was mapped to platlib: /tmp/pip-install-h4586j_l/scipy_d49dfd58e21a4dc0aa66b134666fecba/.mesonpy-dnrdx37d/install/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/version.py (/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/version.py)
            WARNING Could not tell if file was meant for purelib or platlib, so it was mapped to platlib: /tmp/pip-install-h4586j_l/scipy_d49dfd58e21a4dc0aa66b134666fecba/.mesonpy-dnrdx37d/install/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/special/_ufuncs.pyx (/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/special/_ufuncs.pyx)
            WARNING Could not tell if file was meant for purelib or platlib, so it was mapped to platlib: /tmp/pip-install-h4586j_l/scipy_d49dfd58e21a4dc0aa66b134666fecba/.mesonpy-dnrdx37d/install/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/special/_ufuncs_defs.h (/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/special/_ufuncs_defs.h)
            WARNING Could not tell if file was meant for purelib or platlib, so it was mapped to platlib: /tmp/pip-install-h4586j_l/scipy_d49dfd58e21a4dc0aa66b134666fecba/.mesonpy-dnrdx37d/install/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/special/_ufuncs_cxx.pyx (/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/special/_ufuncs_cxx.pyx)
            WARNING Could not tell if file was meant for purelib or platlib, so it was mapped to platlib: /tmp/pip-install-h4586j_l/scipy_d49dfd58e21a4dc0aa66b134666fecba/.mesonpy-dnrdx37d/install/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/special/_ufuncs_cxx.pxd (/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/special/_ufuncs_cxx.pxd)
            WARNING Could not tell if file was meant for purelib or platlib, so it was mapped to platlib: /tmp/pip-install-h4586j_l/scipy_d49dfd58e21a4dc0aa66b134666fecba/.mesonpy-dnrdx37d/install/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/special/_ufuncs_cxx_defs.h (/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/special/_ufuncs_cxx_defs.h)
            WARNING Could not tell if file was meant for purelib or platlib, so it was mapped to platlib: /tmp/pip-install-h4586j_l/scipy_d49dfd58e21a4dc0aa66b134666fecba/.mesonpy-dnrdx37d/install/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/special/_ufuncs.pyi (/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/special/_ufuncs.pyi)
            WARNING Could not tell if file was meant for purelib or platlib, so it was mapped to platlib: /tmp/pip-install-h4586j_l/scipy_d49dfd58e21a4dc0aa66b134666fecba/.mesonpy-dnrdx37d/install/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/special/cython_special.pyx (/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/special/cython_special.pyx)
            WARNING Could not tell if file was meant for purelib or platlib, so it was mapped to platlib: /tmp/pip-install-h4586j_l/scipy_d49dfd58e21a4dc0aa66b134666fecba/.mesonpy-dnrdx37d/install/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/special/cython_special.pxd (/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/special/cython_special.pxd)
            WARNING Could not tell if file was meant for purelib or platlib, so it was mapped to platlib: /tmp/pip-install-h4586j_l/scipy_d49dfd58e21a4dc0aa66b134666fecba/.mesonpy-dnrdx37d/install/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/special/tests/data/boost.npz (/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/special/tests/data/boost.npz)
            WARNING Could not tell if file was meant for purelib or platlib, so it was mapped to platlib: /tmp/pip-install-h4586j_l/scipy_d49dfd58e21a4dc0aa66b134666fecba/.mesonpy-dnrdx37d/install/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/special/tests/data/gsl.npz (/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/special/tests/data/gsl.npz)
            WARNING Could not tell if file was meant for purelib or platlib, so it was mapped to platlib: /tmp/pip-install-h4586j_l/scipy_d49dfd58e21a4dc0aa66b134666fecba/.mesonpy-dnrdx37d/install/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/special/tests/data/local.npz (/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/special/tests/data/local.npz)
            WARNING Could not tell if file was meant for purelib or platlib, so it was mapped to platlib: /tmp/pip-install-h4586j_l/scipy_d49dfd58e21a4dc0aa66b134666fecba/.mesonpy-dnrdx37d/install/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/linalg/cython_blas.pyx (/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/linalg/cython_blas.pyx)
            WARNING Could not tell if file was meant for purelib or platlib, so it was mapped to platlib: /tmp/pip-install-h4586j_l/scipy_d49dfd58e21a4dc0aa66b134666fecba/.mesonpy-dnrdx37d/install/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/linalg/cython_lapack.pyx (/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/linalg/cython_lapack.pyx)
            WARNING Could not tell if file was meant for purelib or platlib, so it was mapped to platlib: /tmp/pip-install-h4586j_l/scipy_d49dfd58e21a4dc0aa66b134666fecba/.mesonpy-dnrdx37d/install/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/linalg/cython_blas.pxd (/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/linalg/cython_blas.pxd)
            WARNING Could not tell if file was meant for purelib or platlib, so it was mapped to platlib: /tmp/pip-install-h4586j_l/scipy_d49dfd58e21a4dc0aa66b134666fecba/.mesonpy-dnrdx37d/install/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/linalg/cython_lapack.pxd (/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/linalg/cython_lapack.pxd)
            WARNING Could not tell if file was meant for purelib or platlib, so it was mapped to platlib: /tmp/pip-install-h4586j_l/scipy_d49dfd58e21a4dc0aa66b134666fecba/.mesonpy-dnrdx37d/install/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/linalg/_blas_subroutines.h (/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/linalg/_blas_subroutines.h)
            WARNING Could not tell if file was meant for purelib or platlib, so it was mapped to platlib: /tmp/pip-install-h4586j_l/scipy_d49dfd58e21a4dc0aa66b134666fecba/.mesonpy-dnrdx37d/install/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/linalg/_lapack_subroutines.h (/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/linalg/_lapack_subroutines.h)
            WARNING Could not tell if file was meant for purelib or platlib, so it was mapped to platlib: /tmp/pip-install-h4586j_l/scipy_d49dfd58e21a4dc0aa66b134666fecba/.mesonpy-dnrdx37d/install/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/linalg/_blas_subroutine_wrappers.f (/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/linalg/_blas_subroutine_wrappers.f)
            WARNING Could not tell if file was meant for purelib or platlib, so it was mapped to platlib: /tmp/pip-install-h4586j_l/scipy_d49dfd58e21a4dc0aa66b134666fecba/.mesonpy-dnrdx37d/install/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/linalg/_lapack_subroutine_wrappers.f (/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/scipy/linalg/_lapack_subroutine_wrappers.f)
            Traceback (most recent call last):
              File "/tmp/pip-build-env-4418iaze/overlay/lib/python3.8-pyston2.3/site-packages/mesonpy/__init__.py", line 339, in _calculate_file_abi_tag_heuristic_posix
                return mesonpy._tags.StableABITag(tag)
              File "/tmp/pip-build-env-4418iaze/overlay/lib/python3.8-pyston2.3/site-packages/mesonpy/_tags.py", line 37, in __init__
                raise ValueError(f'Invalid PEP 3149 stable ABI tag, expecting pattern `{self._REGEX.pattern}`')
            ValueError: Invalid PEP 3149 stable ABI tag, expecting pattern `^abi(?P<abi_number>[0-9]+)$`

            During handling of the above exception, another exception occurred:

            Traceback (most recent call last):
              File "/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
                main()
              File "/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
                json_out['return_val'] = hook(**hook_input['kwargs'])
              File "/users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 152, in prepare_metadata_for_build_wheel
                whl_basename = backend.build_wheel(metadata_directory, config_settings)
              File "/tmp/pip-build-env-4418iaze/overlay/lib/python3.8-pyston2.3/site-packages/mesonpy/__init__.py", line 987, in build_wheel
                return project.wheel(out).name
              File "/tmp/pip-build-env-4418iaze/overlay/lib/python3.8-pyston2.3/site-packages/mesonpy/__init__.py", line 935, in wheel
                wheel = self._wheel_builder.build(self._build_dir)
              File "/tmp/pip-build-env-4418iaze/overlay/lib/python3.8-pyston2.3/site-packages/mesonpy/__init__.py", line 578, in build
                wheel_file = pathlib.Path(directory, f'{self.name}.whl')
              File "/tmp/pip-build-env-4418iaze/overlay/lib/python3.8-pyston2.3/site-packages/mesonpy/__init__.py", line 189, in name
                python_tag=self.python_tag,
              File "/tmp/pip-build-env-4418iaze/overlay/lib/python3.8-pyston2.3/site-packages/mesonpy/__init__.py", line 244, in python_tag
                selected_tag = self._select_abi_tag()
              File "/tmp/pip-build-env-4418iaze/overlay/lib/python3.8-pyston2.3/site-packages/mesonpy/__init__.py", line 375, in _select_abi_tag
                tags = self._files_by_tag()
              File "/tmp/pip-build-env-4418iaze/overlay/lib/python3.8-pyston2.3/site-packages/mesonpy/__init__.py", line 361, in _files_by_tag
                tag = self._calculate_file_abi_tag_heuristic(file)
              File "/tmp/pip-build-env-4418iaze/overlay/lib/python3.8-pyston2.3/site-packages/mesonpy/__init__.py", line 348, in _calculate_file_abi_tag_heuristic
                return self._calculate_file_abi_tag_heuristic_posix(filename)
              File "/tmp/pip-build-env-4418iaze/overlay/lib/python3.8-pyston2.3/site-packages/mesonpy/__init__.py", line 341, in _calculate_file_abi_tag_heuristic_posix
                return mesonpy._tags.InterpreterTag(tag)
              File "/tmp/pip-build-env-4418iaze/overlay/lib/python3.8-pyston2.3/site-packages/mesonpy/_tags.py", line 82, in __init__
                raise NotImplementedError(
            NotImplementedError: Unknown Python implementation: pyston. Please report this to https://github.com/FFY00/mesonpy/issues and include information about the Python distribution you are using.
            [end of output]

        note: This error originates from a subprocess, and is likely not a problem with pip.
      error: metadata-generation-failed

      × Encountered error while generating package metadata.
      ╰─> See above for output.

      note: This is an issue with the package mentioned above, not pip.
      hint: See above for details.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Install scipy

pyston -m pip install scipy --force

Collecting scipy
  Using cached scipy-1.10.0.tar.gz (42.4 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Collecting numpy<1.27.0,>=1.19.5
  Using cached numpy-1.24.2-pyston38-pyston_23_x86_64_linux_gnu-linux_x86_64.whl
Building wheels for collected packages: scipy
  Building wheel for scipy (pyproject.toml) ... done
  Created wheel for scipy: filename=scipy-1.10.0-pyston38-pyston_23_x86_64_linux_gnu-linux_x86_64.whl size=69801459 sha256=9eaba049e053959b9ef850b4197cd56f228da076ce63c0b415b3f10a28d83dfa
  Stored in directory: /users/jean.elbers/.cache/pip/wheels/88/67/32/356a7facb8021804b539189f51f957195ff420791f3c6cd671
Successfully built scipy
Installing collected packages: numpy, scipy
  Attempting uninstall: numpy
    Found existing installation: numpy 1.24.2
    Uninstalling numpy-1.24.2:
      Successfully uninstalled numpy-1.24.2
  Attempting uninstall: scipy
    Found existing installation: scipy 1.10.0
    Uninstalling scipy-1.10.0:
      Successfully uninstalled scipy-1.10.0
Successfully installed numpy-1.24.2 scipy-1.10.0

Ok, retry installing scikit-learn

pyston -m pip install scikit-learn
# same error as before NotImplementedError: Unknown Python implementation: pyston

Ok, try with --no-build-isolation

pyston -m pip install scikit-learn --no-build-isolation
Collecting scikit-learn
  Using cached scikit-learn-1.2.1.tar.gz (7.3 MB)
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [26 lines of output]
      Partial import of sklearn during the build process.
      Traceback (most recent call last):
        File "/tmp/pip-install-gaua6lnd/scikit-learn_e2f302cebe044bda94e34849a6be859e/sklearn/_build_utils/__init__.py", line 27, in _check_cython_version
          import Cython
      ModuleNotFoundError: No module named 'Cython'

Ok, install cython

pyston -m pip install cython
Collecting cython
  Using cached Cython-0.29.33-py2.py3-none-any.whl (987 kB)
Installing collected packages: cython
Successfully installed cython-0.29.33

Retry --no-build-isolation

pyston -m pip install scikit-learn --no-build-isolation
Collecting scikit-learn
  Using cached scikit-learn-1.2.1.tar.gz (7.3 MB)
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: threadpoolctl>=2.0.0 in ./micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages (from scikit-learn) (3.1.0)
Requirement already satisfied: joblib>=1.1.1 in ./micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages (from scikit-learn) (1.2.0)
Requirement already satisfied: scipy>=1.3.2 in ./micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages (from scikit-learn) (1.10.0)
Requirement already satisfied: numpy>=1.17.3 in ./micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages (from scikit-learn) (1.24.2)
Building wheels for collected packages: scikit-learn
  Building wheel for scikit-learn (pyproject.toml) ... done
  Created wheel for scikit-learn: filename=scikit_learn-1.2.1-pyston38-pyston_23_x86_64_linux_gnu-linux_x86_64.whl size=9454170 sha256=e0533d2280646e8e8bcff8f53dfc59caa88675f9e0e9de250c3bf22c59b0ade4
  Stored in directory: /users/jean.elbers/.cache/pip/wheels/6c/87/8f/e70af7f82ed8b6b47b30ed7ab86e48533cc8136cc2a0a166c2
Successfully built scikit-learn
Installing collected packages: scikit-learn
Successfully installed scikit-learn-1.2.1

rgommers · 2023-02-15T11:23:37Z

When you're building a package with compiled code (e.g., depending on numpy) for a conda env, you must use --no-build-isolation. Not doing so is definitely user error, and will get you all sorts of strange issues (unfortunately). So I'll note that pyston -m pip install scipy --force, while it works, is also wrong here.

Why that error gets triggered is unclear, it somehow picks up an old version of meson-python, which will come from scikit-learn somehow trying to depend on an older scipy version.

dnicolodi · 2023-02-15T13:03:26Z

From the build log it also seem to pick an old version of Meson (otherwise the WARNING messages would not be there as that problem is fixed in recent Meson version). It would be nice to have the log lines preceding the ones reported here, I suspect pip is confused by the build dependencies and maybe installing the wrong version of things in the isolated build environment.

Which platform is this? Some pip versions have problems setting up the isolated build environment correctly. This may explain why an old Meson is used. Do you have meson installed on the system? What does meson --version say?

Also, why do you add the --force option to pyston -m pip install scikit-learn?

jelber2 · 2023-02-20T08:33:28Z

I am not sure about the previous parts of the log as it only seems to have the warnings/errors appear before that.

meson --version

-bash: meson: command not found

pip --version 

pip 23.0 from /users/jean.elbers/micromamba/envs/strainy-pyston3/lib/python3.8-pyston2.3/site-packages/pip (python 3.8)

cat /etc/os-release

NAME="CentOS Linux"
VERSION="7 (Core)"

I used --force on accident to be honest.

dnicolodi · 2023-02-20T10:13:55Z

I am not sure about the previous parts of the log as it only seems to have the warnings/errors appear before that.

I don't understand what you are trying to say. Somehow pip arrives to the conclusion that it needs to use old version of meson-python and meson. I would like to understand why. This information is most likely reported by pip in its output (maybe you need to increase the verbosity level above the default). Your problems are caused by pip using old version of the build tools, not by bugs in current versions.

jelber2 · 2023-02-20T10:45:55Z

I am very sorry that I did not look into the log more carefully. I will try again hopefully tomorrow and let you know what I find. Thank you for your time looking into this!

jelber2 · 2023-02-23T11:13:52Z

micromamba create -n strainy-pyston3 -c bioconda -c conda-forge -c pyston samtools=1.14 bcftools=1.14 graphviz pyston
micromamba activate strainy-pyston3

See  https://github.com/mesonbuild/meson-python/issues/142#issuecomment-1431173832 for package versions

pyston -m pip install pygraphviz matplotlib biopython gfapy edlib pysam
pyston -m pip install scikit-learn > pip.log 2>&1 &

pip.log

@dnicolodi Is this enough information? I am not seeing any version numbers for mesonpy here, not sure if traceable back to a specific commit?

jelber2 · 2023-02-23T11:26:01Z

My apologies, I forgot the -vvv part. Here is pyston -m pip -vvv install scikit-learn > pipvvv.log 2>&1 &

When it finishes, I will post a link here: pipvvv.log

dnicolodi · 2023-02-23T11:27:49Z

As I though, you need to increase the verbosity level above the default to have pip log which version of the build dependencies it is installing: pyston -m pip install -v scikit-learn. It would be nice to have the log with the -v options added.

dnicolodi · 2023-02-23T11:28:58Z

My apologies, I forgot the -vvv part

This is going to print way more information than we need, and sifting through the noise is going to take a while, but better than no information.

jelber2 · 2023-02-23T11:34:20Z

@dnicolodi https://www.dropbox.com/s/ctgl82s3kdn0zu7/pipvvv.log Looks like it is using meson-python-0.10.0 at some point?

dnicolodi · 2023-02-23T12:19:43Z

Thanks for the logs. I think the source of the issue is this:

    ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
    matplotlib 3.7.0 requires numpy>=1.20, but you have numpy 1.19.5 which is incompatible.

I don't know why the pip dependency resolver thinks it is a good idea, but from there pip tried to build scipy 4 more times using different versions of scipy and its dependencies. In the last attempt it tries with old meson-python and Meson versions (meson 0.62.2 and meson-python 0.10.0) and it fails because of bugs fixed since these releases in Meson and meson-python.

Most likely, resolving the conflict on the numpy version will make this symptom disappear.

jelber2 · 2023-02-23T13:47:50Z

Thank you very much for your help! Would you like me to try further or do you think this is enough?

dnicolodi · 2023-02-23T14:11:55Z

From our side I think there has never been doubts that things work correctly. The build logs confirm this. If you don't need to install scikit-learn on this system, I don't think there is much more to do. However, you may want to check with the pip developers if this behavior of the dependency resolver is expected. It looks at least suspicious to me.

jelber2 · 2023-02-23T14:45:07Z

Ok thanks!

eli-schwartz · 2023-02-26T16:08:41Z

It looks at least suspicious to me.

https://pip.pypa.io/en/stable/topics/dependency-resolution/#backtracking

The backtracking of choices made during is not unexpected behaviour or a bug. It is part of how dependency resolution for Python packages works.

and

Backtracking reduces the risk that installing a new package will accidentally break an existing installed package, and so reduces the risk that your environment gets messed up. To do this, pip has to do more work, to find out which version of a package is a good candidate to install.

Of course, this doesn't take into consideration the possibility that old versions aren't good candidates because they were buggy. There's really no solution other than, perhaps, yanking all old versions on a regular basis, assuming people never use upper bounds on their meson-python install requirements. Or pip installing with a tightly constrained lower bounds.

dnicolodi · 2023-02-26T17:02:05Z

I'm not questioning backtracking, just the choice of packages to backtrack. I haven't analyzed the log in detail, but I think that the source of the issue is the numpy version available and the one required by matplotlib and it seems that pip though that rebuilding the same version of scipy with older and older versions of meson and meson-python may have fixed that. I'm surprised that older version of meson-python and meson were tried before trying older versions of scipy. But, as I wrote, I didn't analyze the log in detail and neither I tried to build the dependency graph.

eli-schwartz · 2023-02-26T17:53:40Z

just the choice of packages to backtrack

"try random stuff and see if it sticks". :P

I suppose the logic is that the backtracker assumes that you want the latest version of the package you asked for, which can be built -- so using older versions of its dependencies could yield better results than older versions of that very package.

dnicolodi · 2023-02-26T18:13:46Z

The package that was asked for is scikit-learn here :)

eli-schwartz · 2023-02-26T18:23:56Z

Oh hmm, good point, that spoils that theory... dunno then, the whole backtracking logic seems very... mysterious.

rgommers · 2023-02-26T19:24:52Z

There's still work to be done in pip to make this behave better. IIRC the "if a build fails, then don't retry at all" isn't implemented yet - or if it is, then in a very recent version. Backtracking when dependency conflicts are encountered makes sense. Retrying over and over when builds are failing less so.

FFY00 added the enhancement New feature or request label Sep 14, 2022

belm0 mentioned this issue Nov 3, 2022

ENH: delegate computing wheel tags to the packaging module #191

Closed

FFY00 closed this as completed in 9e84bd9 Nov 16, 2022

Unknown Python implementation: pyston (cannot install SciPy >= 1.9.0 on Pyston) #142

Unknown Python implementation: pyston (cannot install SciPy >= 1.9.0 on Pyston) #142

Comments

PLPeeters commented Sep 14, 2022

FFY00 commented Sep 14, 2022

PLPeeters commented Sep 15, 2022

rgommers commented Sep 27, 2022

FFY00 commented Sep 28, 2022

kmod commented Sep 28, 2022

undingen commented Sep 29, 2022

jelber2 commented Oct 19, 2022

dnicolodi commented Nov 2, 2022

belm0 commented Nov 3, 2022 • edited Loading

dnicolodi commented Nov 3, 2022

rgommers commented Nov 3, 2022

belm0 commented Nov 3, 2022

dnicolodi commented Nov 3, 2022

belm0 commented Nov 3, 2022

dnicolodi commented Nov 3, 2022

belm0 commented Nov 3, 2022

dnicolodi commented Nov 3, 2022

belm0 commented Nov 3, 2022 • edited Loading

belm0 commented Nov 3, 2022

rgommers commented Nov 4, 2022

jelber2 commented Feb 14, 2023

dnicolodi commented Feb 14, 2023

rgommers commented Feb 14, 2023

jelber2 commented Feb 14, 2023

dnicolodi commented Feb 14, 2023

jelber2 commented Feb 14, 2023

jelber2 commented Feb 15, 2023 • edited Loading

rgommers commented Feb 15, 2023

dnicolodi commented Feb 15, 2023

jelber2 commented Feb 20, 2023

dnicolodi commented Feb 20, 2023

jelber2 commented Feb 20, 2023

jelber2 commented Feb 23, 2023 • edited Loading

jelber2 commented Feb 23, 2023 • edited Loading

dnicolodi commented Feb 23, 2023

dnicolodi commented Feb 23, 2023

jelber2 commented Feb 23, 2023 • edited Loading

dnicolodi commented Feb 23, 2023

jelber2 commented Feb 23, 2023

dnicolodi commented Feb 23, 2023

jelber2 commented Feb 23, 2023

eli-schwartz commented Feb 26, 2023 • edited Loading

dnicolodi commented Feb 26, 2023

eli-schwartz commented Feb 26, 2023

dnicolodi commented Feb 26, 2023

eli-schwartz commented Feb 26, 2023

rgommers commented Feb 26, 2023

belm0 commented Nov 3, 2022 •

edited

Loading

belm0 commented Nov 3, 2022 •

edited

Loading

jelber2 commented Feb 15, 2023 •

edited

Loading

jelber2 commented Feb 23, 2023 •

edited

Loading

jelber2 commented Feb 23, 2023 •

edited

Loading

jelber2 commented Feb 23, 2023 •

edited

Loading

eli-schwartz commented Feb 26, 2023 •

edited

Loading