Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mamba brings pypy instead of CPython when creating env #658

Closed
Tracked by #1043
gabrielcnr opened this issue Jan 15, 2021 · 18 comments
Closed
Tracked by #1043

mamba brings pypy instead of CPython when creating env #658

gabrielcnr opened this issue Jan 15, 2021 · 18 comments
Labels
status::need-triage New feature proposal that have not been reviewed. where::libsolv

Comments

@gabrielcnr
Copy link

I have here what it seems to be related to the issue #81

When using mamba to create an environment, it keeps pulling pypy, which is not part of the given specs. When using pure conda that does not happen.

I tested this on Mac and Linux, and both presented the same behaviour.

Creating an env with:

mamba create -n foobar "cachetools" "dataclasses" "diskcache>=4,<5" "humanize" "matplotlib>=3.1.3,<3.2" "numpy>=1.12.1" "pandas<1" "pydantic " "pytest" "pytest-mock" "pytest-datadir" "python>=3.6.10,<3.7" "python-dateutil" "pytz" "requests-mock>=1.7,<1.8" "ujson" "lxml>=3.8.0" "beautifulsoup4>=4.8 .0,<4.9"

Gives:

Looking for: ['cachetools', 'dataclasses', "diskcache[version='>=4,<5']", 'humanize', "matplotlib[version='>=3.1.3,<3.2']", "numpy[version='>=1.12.1']", "pandas[version='<1']", 'pydantic', 'pytest', 'pytest-mock', 'pytest-datadir', "python[version='>=3.6.10,<3.7']", 'python-dateutil', 'pytz', "requests-mock[version='>=1.7,<1.8']", 'ujson', "lxml[version='>=3.8.0']", "beautifulsoup4[version='>=4.8.0,<4.9']"]

conda-forge/osx-64       Using cache
conda-forge/noarch       Using cache
main/noarch              [====================] (00m:00s) No change
main/osx-64              [====================] (00m:00s) No change
Transaction

  Prefix: /Users/greis/conda/envs/boa-dev/envs/foobar

  Updating specs:

   - cachetools
   - dataclasses
   - diskcache[version='>=4,<5']
   - humanize
   - matplotlib[version='>=3.1.3,<3.2']
   - numpy[version='>=1.12.1']
   - pandas[version='<1']
   - pydantic
   - pytest
   - pytest-mock
   - pytest-datadir
   - python[version='>=3.6.10,<3.7']
   - python-dateutil
   - pytz
   - requests-mock[version='>=1.7,<1.8']
   - ujson
   - lxml[version='>=3.8.0']
   - beautifulsoup4[version='>=4.8.0,<4.9']


  Package               Version  Build              Channel                  Size
───────────────────────────────────────────────────────────────────────────────────
  Install:
───────────────────────────────────────────────────────────────────────────────────

  attrs                  20.3.0  pyhd3eb1b0_0       main/noarch             42 KB
  beautifulsoup4          4.8.2  py36_0             main/osx-64            161 KB
  brotlipy                0.7.0  py36h9ed2024_1003  main/osx-64            359 KB
  bzip2                   1.0.8  h1de35cc_0         main/osx-64            104 KB
  ca-certificates     2020.12.8  hecd8cb5_0         main/osx-64            Cached
  cachetools              4.2.0  pyhd3eb1b0_0       main/noarch             13 KB
  certifi             2020.12.5  py36hecd8cb5_0     main/osx-64            143 KB
  cffi                   1.14.4  py36h2125817_0     main/osx-64            217 KB
  chardet                 4.0.0  py36hecd8cb5_1003  main/osx-64            219 KB
  cryptography            3.3.1  py36hbcfaee0_0     main/osx-64            615 KB
  cycler                 0.10.0  py36hfc81398_0     main/osx-64             13 KB
  dataclasses               0.7  py36_0             main/osx-64             30 KB
  diskcache               4.1.0  pyh9f0ad1d_0       conda-forge/noarch      37 KB
  expat                  2.2.10  hb1e8313_2         main/osx-64            Cached
  freetype               2.10.4  ha233b18_0         main/osx-64            891 KB
  gdbm                     1.18  h8a0c380_2         conda-forge/osx-64     131 KB
  humanize                3.2.0  pyhd3eb1b0_0       main/noarch             39 KB
  icu                      58.2  h0a44026_3         main/osx-64             23 MB
  idna                     2.10  py_0               main/noarch            Cached
  importlib-metadata      3.4.0  py36h5dafb3c_0     conda-forge/osx-64      20 KB
  importlib_metadata      3.4.0  hd8ed1ab_0         conda-forge/noarch       3 KB
  iniconfig               1.1.1  py_0               main/noarch              8 KB
  kiwisolver              1.3.1  py36hb084ce4_1     conda-forge/osx-64      57 KB
  libblas                 3.9.0  7_openblas         conda-forge/osx-64      11 KB
  libcblas                3.9.0  7_openblas         conda-forge/osx-64      11 KB
  libcxx                 11.0.0  h4c3b8ed_1         conda-forge/osx-64     Cached
  libffi                    3.3  hb1e8313_2         main/osx-64            Cached
  libgfortran             5.0.0  9_3_0_h6c81a4c_16  conda-forge/osx-64      19 KB
  libgfortran5            9.3.0  h6c81a4c_16        conda-forge/osx-64       2 MB
  libiconv                 1.16  h1de35cc_0         main/osx-64            Cached
  liblapack               3.9.0  7_openblas         conda-forge/osx-64      11 KB
  libopenblas            0.3.12  openmp_h54245bb_1  conda-forge/osx-64       9 MB
  libpng                 1.6.37  ha441bb4_0         main/osx-64            325 KB
  libxml2                2.9.10  h7cdb67c_3         main/osx-64              1 MB
  libxslt                1.1.34  h83b36ba_0         main/osx-64            534 KB
  llvm-openmp            11.0.0  h73239a0_1         conda-forge/osx-64     268 KB
  lxml                    4.6.2  py36h26b266a_0     main/osx-64              1 MB
  matplotlib              3.1.3  py36_0             main/osx-64             20 KB
  matplotlib-base         3.1.3  py36h9aa3819_0     main/osx-64              7 MB
  more-itertools          8.6.0  pyhd3eb1b0_0       main/noarch             38 KB
  ncurses                   6.2  h0a44026_1         main/osx-64            Cached
  numpy                  1.19.5  py36h9faff70_1     conda-forge/osx-64       5 MB
  openssl                1.1.1i  h9ed2024_0         main/osx-64            Cached
  packaging                20.8  pyhd3eb1b0_0       main/noarch            Cached
  pandas                 0.25.3  py36h0a44026_0     main/osx-64             10 MB
  pip                    20.3.3  py36hecd8cb5_0     main/osx-64              2 MB
  pluggy                 0.13.1  py36_0             main/osx-64             32 KB
  py                     1.10.0  pyhd3eb1b0_0       main/noarch             74 KB
  pycparser                2.20  py_2               main/noarch            Cached
  pydantic                1.7.3  py36h9ed2024_0     main/osx-64              2 MB
  pyopenssl              20.0.1  pyhd3eb1b0_1       main/noarch            Cached
  pyparsing               2.4.7  py_0               main/noarch            Cached
  pypy3.6                 7.3.3  h41cf889_3         conda-forge/osx-64      26 MB
  pysocks                 1.7.1  py36_0             main/osx-64             30 KB
  pytest                  6.2.1  py36hecd8cb5_0     main/osx-64            429 KB
  pytest-datadir          1.3.1  py_0               conda-forge/noarch       7 KB
  pytest-mock             3.5.1  pyhd3eb1b0_0       main/noarch             20 KB
  python                 3.6.12  5_73_pypy          conda-forge/osx-64       5 KB
  python-dateutil         2.8.1  py_0               main/noarch            224 KB
  python_abi                3.6  1_pypy36_pp73      conda-forge/osx-64       4 KB
  pytz                   2020.5  pyhd3eb1b0_0       main/noarch            Cached
  readline                  8.0  h1de35cc_0         main/osx-64            Cached
  requests               2.25.1  pyhd3eb1b0_0       main/noarch            Cached
  requests-mock           1.7.0  py_0               conda-forge/noarch      20 KB
  setuptools             51.1.2  py36hecd8cb5_3     main/osx-64            906 KB
  six                    1.15.0  py36hecd8cb5_0     main/osx-64             27 KB
  soupsieve                 2.1  pyhd3eb1b0_0       main/noarch             30 KB
  sqlite                 3.34.0  h17101e1_0         conda-forge/osx-64     Cached
  tk                     8.6.10  hb0a8c7a_0         main/osx-64            Cached
  toml                   0.10.2  pyhd8ed1ab_0       conda-forge/noarch      18 KB
  tornado                   6.1  py36h9ed2024_0     main/osx-64            642 KB
  typing_extensions     3.7.4.3  py_0               main/noarch             29 KB
  ujson                   4.0.1  py36hb1e8313_0     main/osx-64             47 KB
  urllib3                1.26.2  pyhd3eb1b0_0       main/noarch            Cached
  wheel                  0.36.2  pyhd3eb1b0_0       main/noarch            Cached
  xz                      5.2.5  h1de35cc_0         main/osx-64            Cached
  zipp                    3.4.0  pyhd3eb1b0_0       main/noarch             15 KB
  zlib                   1.2.11  h1de35cc_3         main/osx-64            Cached

  Summary:

  Install: 78 packages

  Total download: 94 MB

As you can see from above, the pypy-related packages:

pypy3.6                 7.3.3  h41cf889_3         conda-forge/osx-64      26 MB
python                 3.6.12  5_73_pypy          conda-forge/osx-64       5 KB

environment

My environment is (truncated below):

boa                       0.3.3                     dev_0    <develop>
conda                     4.9.2            py38hecd8cb5_0    main
conda-build               3.21.3           py38hecd8cb5_0    main
libsolv                   0.7.16               hab81aa3_0    conda-forge
mamba                     0.7.6            py38he6f4b85_0    conda-forge
python                    3.8.5                h26836e1_1    main

@wolfv @SylvainCorlay - I tried to debug/investigate the problem but I confess I got a bit lost and realised that I wouldn't have the time now., so I'm sorry I can't give more details here at the moment. But I was wondering if there could be an easy workaround for this problem? For example, maybe some kind of version constraint hint we can give to instruct the solver to ignore the *pypy* build strings?

@marcelotrevisani
Copy link
Contributor

I tested it on my local and it works fine for Linux and mamba=0.4.4

@marcelotrevisani
Copy link
Contributor

Well, I also tried with 0.7.8 and it worked fine... that is weird

@marcelotrevisani
Copy link
Contributor

As a workaround you can write python as:

"python=3.6.*=*_cpython"

@gabrielcnr
Copy link
Author

I reinstalled mamba and now I'm on mamba=0.7.8 and I continue to see the same problem on both Mac and Linux.

@gabrielcnr
Copy link
Author

gabrielcnr commented Jan 15, 2021

I have further information here, after a bit more investigation.
I noticed if I change the specs that I pass to mamba to consider only python instead of python>=3.6.10,<3.7", then it correctly picks up the CPython python. If try to pass python=3.7.* or python=3.8.* it also works fine. But if I try to pass python=3.6.* then it picks up the pypy build. So, I guess it's something related to python 3.6.

The only way that I and @marcelotrevisani managed to make mamba to pick up the CPython version for Python 3.6 was by specifying "python>=3.6.10,<3.7=*_cpython"

@SylvainCorlay
Copy link
Member

@gabrielcnr, could you please report the version of libsolv that you have?
This is related to openSUSE/libsolv#395 which was only included in libsolv 0.7.15.

@gabrielcnr
Copy link
Author

@SylvainCorlay , I've got libsolv=0.7.16

More specifically:

libsolv                   0.7.16               hab81aa3_0    conda-forge

@wolfv
Copy link
Member

wolfv commented Jan 17, 2021

@gabrielcnr I suppose you're running natively on OSX, right?

When testing on Linux, using

CONDA_SUBDIR=osx-64 mamba create -n foobar "cachetools" "dataclasses" "diskcache>=4,<5" "humanize" "matplotlib>=3.1.3,<3.2" "numpy>=1.12.1" "pandas<1" "pydantic " "pytest" "pytest-mock" "pytest-datadir" "python>=3.6.10,<3.7" "python-dateutil" "pytz" "requests-mock>=1.7,<1.8" "ujson" "lxml>=3.8.0" "beautifulsoup4>=4.8 .0,<4.9"

mamba correctly resolves to CPython. I wonder if this is an issue on the OS X build of libsolv ... that would be interesting :)

@wolfv
Copy link
Member

wolfv commented Jan 17, 2021

Hmm, I tried locally on OS X (intel) and your command also properly takes cpython.

What happens if you do not use main as additional channel? It's discouraged, and they never did the python_abi patches that conda-forge has to support both pypy and cpython.

Looking for: ['cachetools', 'dataclasses', "diskcache[version='>=4,<5']", 'humanize', "matplotlib[version='>=3.1.3,<3.2']", "numpy[version='>=1.12.1']", "pandas[version='<1']", 'pydantic', 'pytest', 'pytest-mock', 'pytest-datadir', "python[version='>=3.6.10,<3.7']", 'python-dateutil', 'pytz', "requests-mock[version='>=1.7,<1.8']", 'ujson', "lxml[version='>=3.8.0']", "beautifulsoup4[version='>=4.8.0,<4.9']"]

conda-forge/osx-64       Using cache
conda-forge/noarch       Using cache
Users/wolfv/mini38/conda [====================] (00m:00s) Done
Users/wolfv/mini38/conda [====================] (00m:00s) Done
robostack/osx-64         [====================] (00m:00s) No change
robostack/noarch         [====================] (00m:00s) No change
Transaction

  Prefix: /Users/wolfv/mini38/envs/foobar

  Updating specs:

   - cachetools
   - dataclasses
   - diskcache[version='>=4,<5']
   - humanize
   - matplotlib[version='>=3.1.3,<3.2']
   - numpy[version='>=1.12.1']
   - pandas[version='<1']
   - pydantic
   - pytest
   - pytest-mock
   - pytest-datadir
   - python[version='>=3.6.10,<3.7']
   - python-dateutil
   - pytz
   - requests-mock[version='>=1.7,<1.8']
   - ujson
   - lxml[version='>=3.8.0']
   - beautifulsoup4[version='>=4.8.0,<4.9']


  Package               Version  Build               Channel                  Size
────────────────────────────────────────────────────────────────────────────────────
  Install:
────────────────────────────────────────────────────────────────────────────────────

  attrs                  20.3.0  pyhd3deb0d_0        conda-forge/noarch     Cached
  beautifulsoup4          4.8.2  py36h9f0ad1d_1      conda-forge/osx-64     157 KB
  brotlipy                0.7.0  py36ha3de816_1001   conda-forge/osx-64     357 KB
  ca-certificates     2020.12.5  h033912b_0          conda-forge/osx-64     Cached
  cachetools              4.1.1  py_0                conda-forge/noarch     Cached
  certifi             2020.12.5  py36h79c6626_1      conda-forge/osx-64     143 KB
  cffi                   1.14.4  py36hfe0777f_1      conda-forge/osx-64     217 KB
  chardet                 4.0.0  py36h79c6626_1      conda-forge/osx-64     219 KB
  cryptography            3.3.1  py36hd858a44_1      conda-forge/osx-64     620 KB
  cycler                 0.10.0  py36_0              conda-forge/osx-64      14 KB
  dataclasses               0.7  py36_0              conda-forge/osx-64      28 KB
  diskcache               4.1.0  pyh9f0ad1d_0        conda-forge/noarch      37 KB
  freetype               2.10.4  h4cff582_1          conda-forge/osx-64     890 KB
  humanize                3.2.0  pyhd8ed1ab_0        conda-forge/noarch      38 KB
  icu                      68.1  h74dc148_0          conda-forge/osx-64     Cached
  idna                     2.10  pyh9f0ad1d_0        conda-forge/noarch     Cached
  importlib-metadata      3.4.0  py36h79c6626_0      conda-forge/osx-64      20 KB
  importlib_metadata      3.4.0  hd8ed1ab_0          conda-forge/noarch       3 KB
  iniconfig               1.1.1  pyh9f0ad1d_0        conda-forge/noarch       8 KB
  kiwisolver              1.3.1  py36h615c93b_1      conda-forge/osx-64      59 KB
  libblas                 3.9.0  7_openblas          conda-forge/osx-64      11 KB
  libcblas                3.9.0  7_openblas          conda-forge/osx-64      11 KB
  libcxx                 11.0.0  h4c3b8ed_1          conda-forge/osx-64     Cached
  libffi                    3.3  h046ec9c_2          conda-forge/osx-64     Cached
  libgfortran             5.0.0  9_3_0_h6c81a4c_16   conda-forge/osx-64      19 KB
  libgfortran5            9.3.0  h6c81a4c_16         conda-forge/osx-64       2 MB
  libiconv                 1.16  haf1e3a3_0          conda-forge/osx-64     Cached
  liblapack               3.9.0  7_openblas          conda-forge/osx-64      11 KB
  libopenblas            0.3.12  openmp_h54245bb_1   conda-forge/osx-64     Cached
  libpng                 1.6.37  hb0a8c7a_2          conda-forge/osx-64     Cached
  libxml2                2.9.10  h802ca4f_3          conda-forge/osx-64       1 MB
  libxslt                1.1.33  h5739fc3_2          conda-forge/osx-64     488 KB
  llvm-openmp            11.0.1  h7c73e74_0          conda-forge/osx-64     268 KB
  lxml                    4.6.2  py36ha579a0c_1      conda-forge/osx-64       1 MB
  matplotlib              3.1.3  py36_0              conda-forge/osx-64       7 KB
  matplotlib-base         3.1.3  py36h11da6c2_0      conda-forge/osx-64       7 MB
  more-itertools          8.6.0  pyhd8ed1ab_0        conda-forge/noarch      37 KB
  ncurses                   6.2  h2e338ed_4          conda-forge/osx-64     Cached
  numpy                  1.19.5  py36h08dc641_1      conda-forge/osx-64       5 MB
  openssl                1.1.1i  h35c211d_0          conda-forge/osx-64     Cached
  packaging                20.8  pyhd3deb0d_0        conda-forge/noarch      34 KB
  pandas                 0.25.3  py36h4f17bb1_0      conda-forge/osx-64      10 MB
  pip                    20.3.3  pyhd8ed1ab_0        conda-forge/noarch       1 MB
  pluggy                 0.13.1  py36h79c6626_4      conda-forge/osx-64      29 KB
  py                     1.10.0  pyhd3deb0d_0        conda-forge/noarch      73 KB
  pycparser                2.20  pyh9f0ad1d_2        conda-forge/noarch     Cached
  pydantic                1.7.3  py36h7b4949f_0      conda-forge/osx-64       2 MB
  pyopenssl              20.0.1  pyhd8ed1ab_0        conda-forge/noarch      48 KB
  pyparsing               2.4.7  pyh9f0ad1d_0        conda-forge/noarch     Cached
  pysocks                 1.7.1  py36h79c6626_3      conda-forge/osx-64      27 KB
  pytest                  6.2.1  py36h79c6626_1      conda-forge/osx-64     429 KB
  pytest-datadir          1.3.1  py_0                conda-forge/noarch       7 KB
  pytest-mock             3.5.1  pyhd8ed1ab_0        conda-forge/noarch      19 KB
  python                 3.6.12  h6c3b2c9_0_cpython  conda-forge/osx-64      21 MB
  python-dateutil         2.8.1  py_0                conda-forge/noarch     Cached
  python_abi                3.6  1_cp36m             conda-forge/osx-64     Cached
  pytz                   2020.5  pyhd8ed1ab_0        conda-forge/noarch     244 KB
  readline                  8.0  h0678c8f_2          conda-forge/osx-64     Cached
  requests               2.25.1  pyhd3deb0d_0        conda-forge/noarch      51 KB
  requests-mock           1.7.0  py_0                conda-forge/noarch      20 KB
  setuptools             49.6.0  py36h79c6626_3      conda-forge/osx-64     916 KB
  six                    1.15.0  pyh9f0ad1d_0        conda-forge/noarch     Cached
  soupsieve               2.0.1  py36h9f0ad1d_0      conda-forge/osx-64      57 KB
  sqlite                 3.34.0  h17101e1_0          conda-forge/osx-64     Cached
  tk                     8.6.10  hb0a8c7a_1          conda-forge/osx-64     Cached
  toml                   0.10.2  pyhd8ed1ab_0        conda-forge/noarch      18 KB
  tornado                   6.1  py36h20b66c6_1      conda-forge/osx-64     643 KB
  typing_extensions     3.7.4.3  py_0                conda-forge/noarch     Cached
  ujson                   4.0.1  py36h22858aa_2      conda-forge/osx-64      47 KB
  urllib3                1.26.2  pyhd8ed1ab_0        conda-forge/noarch     Cached
  wheel                  0.36.2  pyhd3deb0d_0        conda-forge/noarch      31 KB
  xz                      5.2.5  haf1e3a3_1          conda-forge/osx-64     Cached
  zipp                    3.4.0  py_0                conda-forge/noarch     Cached
  zlib                   1.2.11  h7795811_1010       conda-forge/osx-64     Cached

  Summary:

  Install: 74 packages

  Total download: 56 MB

────────────────────────────────────────────────────────────────────────────────────

Now I don't really know any further.

@gabrielcnr
Copy link
Author

@wolfv let me try here, but I had the same problem originally on Linux (RHEL 7) at work...

@gabrielcnr
Copy link
Author

I can confirm that I'm seeing this both on Mac and on Linux. Just tried here and the same happened... Do you need more details about my environment? Please let me know and I can provide more info.

@gabrielcnr
Copy link
Author

@wolfv - If add -c conda-forge at the end of the command to create the env, then it resolves pulling CPython... so, the hunch was good... :-)

@wolfv
Copy link
Member

wolfv commented Jan 17, 2021

Yeah, IMHO conda-forge and main are becoming more and more incompatible. It's like using fedora and Ubuntu packages at the same time.

One should at least use strict channel priority, better yet completely remove Main on Linux and OS X.

The defaults channel never modified their python packages to be compatible with cpython/pypy

@SylvainCorlay
Copy link
Member

I think that it is time communicating more about using mambaforge / miniforge, rather than miniconda.

@marcelotrevisani
Copy link
Contributor

marcelotrevisani commented Jan 17, 2021

The defaults channel never modified their python packages to be compatible with cpython/pypy

That is right, but, it is because the main don't care about pypy, but the python on the main channel and the python with CPython on conda-forge are basically the same

So, it is not being incompatible, the fact that it is doing extensions to support other python implementations, that is a big difference

@m-rossi
Copy link

m-rossi commented May 2, 2021

I have experienced a similar observation in my weekly test-pipeline. Somehow the Python 3.7 tests on macOS and Linux failed. The cause? The install process in an already existing environment updated the Python 3.7 version to the PyPy 3.7 version, see this github-actions log as an example: https://github.com/m-rossi/jupyter-docx-bundler/runs/2486013432?check_suite_focus=true#step:4:128

After adding the workaround

As a workaround you can write python as:

"python=3.6.*=*_cpython"

from @marcelotrevisani (thanks!) it works as expected. However it was not necessary before to define the Python for an install process for an already existing environment.

@adriendelsalle adriendelsalle added the status::need-triage New feature proposal that have not been reviewed. label Jun 4, 2021
@wolfv
Copy link
Member

wolfv commented Jun 30, 2021

This will be fixed by openSUSE/libsolv#457

@wolfv wolfv mentioned this issue Jun 30, 2021
17 tasks
@wolfv
Copy link
Member

wolfv commented Jun 30, 2021

Can you test this again with the latest libsolv installed? I've pushed this patch to the conda-forge build of libsolv: openSUSE/libsolv#457

@wolfv wolfv closed this as completed Jun 30, 2021
orbeckst added a commit to Becksteinlab/GromacsWrapper that referenced this issue Sep 15, 2023
- work around some weird installation issues that are probably due to
  the fact that we need to pull GROMACS from bioconda in some instances
- (1) For macOS Python 2.7, numkit gets uninstalled: need to keep it
  explicitly.
- (2) For Python 3.9, mamba wants to replace cpython with pypy and that
  leads to all kinds of problems (similar to what was reported in
  mamba-org/mamba#658 ). We explicitly request
  the cpython version of Python 3.9 when installing GROMACS.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
status::need-triage New feature proposal that have not been reviewed. where::libsolv
Projects
None yet
Development

No branches or pull requests

6 participants