Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove cuda from default virtual packages? #426

Closed
2 tasks done
minrk opened this issue Jun 2, 2023 · 6 comments · Fixed by #428
Closed
2 tasks done

Remove cuda from default virtual packages? #426

minrk opened this issue Jun 2, 2023 · 6 comments · Fixed by #428

Comments

@minrk
Copy link
Contributor

minrk commented Jun 2, 2023

Checklist

  • I added a descriptive title
  • I searched open reports and couldn't find a duplicate

What happened?

I noticed while installing pytorch-cpu that it pulled in the cuda variant of libhwloc and thereby cudatoolkit, doubling the size of my image. I tracked it down to the default virtual package spec assuming all machines are likely to have cuda by default.

I was able to solve it with a custom virtual packages spec (capturing the virtual packages from conda info in the base image), but it seems to make more sense to me for cuda to be opt-in instead of opt-out, since it's unvailable more often than not, and the cost of an incorrect assumption of its presence is high (massive size increase, non-working packages), while the cost of missing it is low (reduced performance or informative, immediate error if cuda is actually required to install). Or is there a consideration I'm missing?

Conda Info

active environment : base
    active env location : /opt/conda
            shell level : 1
       user config file : /home/user/.condarc
 populated config files : /opt/conda/.condarc
          conda version : 23.3.1
    conda-build version : not installed
         python version : 3.10.11.final.0
       virtual packages : __archspec=1=x86_64
                          __glibc=2.35=0
                          __linux=5.15.0=0
                          __unix=0=0
       base environment : /opt/conda  (writable)
      conda av data dir : /opt/conda/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://conda.anaconda.org/minrk/linux-64
                          https://conda.anaconda.org/minrk/noarch
          package cache : /opt/conda/pkgs
                          /home/user/.conda/pkgs
       envs directories : /opt/conda/envs
                          /home/user/.conda/envs
               platform : linux-64
             user-agent : conda/23.3.1 requests/2.31.0 CPython/3.10.11 Linux/5.15.0-1031-gcp ubuntu/22.04.2 glibc/2.35
                UID:GID : 1000:100
             netrc file : None
           offline mode : False

Conda Config

==> /opt/conda/.condarc <==
auto_update_conda: False
channels:
  - conda-forge
show_channel_urls: True

Conda list

# packages in environment at /Users/minrk/conda:
#
# Name                    Version                   Build  Channel
aiofiles                  22.1.0             pyhd8ed1ab_0    conda-forge
aiosqlite                 0.18.0             pyhd8ed1ab_0    conda-forge
alembic                   1.9.3              pyhd8ed1ab_0    conda-forge
altair                    4.2.2              pyhd8ed1ab_0    conda-forge
anyio                     3.6.2              pyhd8ed1ab_0    conda-forge
appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
appnope                   0.1.3              pyhd8ed1ab_0    conda-forge
argon2-cffi               21.3.0             pyhd8ed1ab_0    conda-forge
argon2-cffi-bindings      21.2.0          py311he2be06e_3    conda-forge
arrow-cpp                 12.0.0           hce30654_1_cpu    conda-forge
asttokens                 2.2.1              pyhd8ed1ab_0    conda-forge
aws-c-auth                0.6.26               h0c2019c_6    conda-forge
aws-c-cal                 0.5.26               h9c01cc2_0    conda-forge
aws-c-common              0.8.17               hb547adb_0    conda-forge
aws-c-compression         0.2.16               h7a7fafa_6    conda-forge
aws-c-event-stream        0.2.20               heb79caa_6    conda-forge
aws-c-http                0.7.7                h5140e40_3    conda-forge
aws-c-io                  0.13.21              hc2875bd_3    conda-forge
aws-c-mqtt                0.8.6               h34492a6_15    conda-forge
aws-c-s3                  0.2.8                h28047f4_4    conda-forge
aws-c-sdkutils            0.1.9                h7a7fafa_1    conda-forge
aws-checksums             0.1.14               h7a7fafa_6    conda-forge
aws-crt-cpp               0.19.9               h1ca6c90_5    conda-forge
aws-sdk-cpp               1.10.57             hc8901a9_10    conda-forge
babel                     2.11.0             pyhd8ed1ab_0    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                pyhd8ed1ab_3    conda-forge
backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
beautifulsoup4            4.11.2             pyha770c72_0    conda-forge
black                     23.3.0                   pypi_0    pypi
bleach                    6.0.0              pyhd8ed1ab_0    conda-forge
blinker                   1.5                pyhd8ed1ab_0    conda-forge
boa                       0.14.0             pyhd8ed1ab_0    conda-forge
boltons                   23.0.0             pyhd8ed1ab_0    conda-forge
boolean.py                3.7                        py_0    conda-forge
brotli                    1.0.9                h1a8c8d9_8    conda-forge
brotli-bin                1.0.9                h1a8c8d9_8    conda-forge
brotlipy                  0.7.0           py311he2be06e_1005    conda-forge
bzip2                     1.0.8                h3422bc3_4    conda-forge
c-ares                    1.18.1               h3422bc3_0    conda-forge
ca-certificates           2023.5.7             hf0a4a13_0    conda-forge
cachecontrol              0.13.0                   pypi_0    pypi
cachetools                5.3.1                    pypi_0    pypi
cachy                     0.3.0                    pypi_0    pypi
cctools                   973.0.1             hcbb26d4_11    conda-forge
cctools_osx-arm64         973.0.1             hef52d2f_11    conda-forge
certifi                   2023.5.7           pyhd8ed1ab_0    conda-forge
cffi                      1.15.1          py311hae827db_3    conda-forge
cfgv                      3.3.1                    pypi_0    pypi
chardet                   5.1.0           py311h267d04e_0    conda-forge
charset-normalizer        2.1.1              pyhd8ed1ab_0    conda-forge
cli-ui                    0.17.2                   pypi_0    pypi
click                     8.1.3                    pypi_0    pypi
click-default-group       1.2.2                    pypi_0    pypi
clikit                    0.6.2                    pypi_0    pypi
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
comm                      0.1.2              pyhd8ed1ab_0    conda-forge
conda                     23.3.1          py311h267d04e_0    conda-forge
conda-build               3.23.3          py311h267d04e_1    conda-forge
conda-forge-pinning       2023.02.16.23.42.32      hd8ed1ab_0    conda-forge
conda-libmamba-solver     23.1.0             pyhd8ed1ab_0    conda-forge
conda-lock                2.0.0                    pypi_0    pypi
conda-package-handling    2.0.2              pyh38be061_0    conda-forge
conda-package-streaming   0.7.0              pyhd8ed1ab_1    conda-forge
conda-smithy              3.23.1             pyhd8ed1ab_0    conda-forge
conda-tree                1.1.0              pyhd8ed1ab_0    conda-forge
contextlib2               21.6.0                   pypi_0    pypi
contourpy                 1.0.7           py311hd6ee22a_0    conda-forge
crashtest                 0.3.1                    pypi_0    pypi
cryptography              41.0.0          py311h5fb2c35_0    conda-forge
curl                      8.0.1                heffe338_0    conda-forge
cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
darker                    1.7.1                    pypi_0    pypi
dataclasses               0.8                pyhc8e2a94_3    conda-forge
debugpy                   1.6.7           py311ha397e9f_0    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
deprecated                1.2.13             pyh6c4a22f_0    conda-forge
distlib                   0.3.6                    pypi_0    pypi
docker                    6.1.3                    pypi_0    pypi
docker-image-cleaner      1.0.0b3                  pypi_0    pypi
docopt                    0.6.2                    pypi_0    pypi
docutils                  0.20.1                   pypi_0    pypi
ensureconda               1.4.3                    pypi_0    pypi
entrypoints               0.4                pyhd8ed1ab_0    conda-forge
executing                 1.2.0              pyhd8ed1ab_0    conda-forge
expat                     2.5.0                hb7217d7_0    conda-forge
filelock                  3.12.0                   pypi_0    pypi
flit-core                 3.8.0              pyhd8ed1ab_0    conda-forge
fmt                       9.1.0                hffc8910_0    conda-forge
fonttools                 4.39.4          py311heffc1b2_0    conda-forge
freetype                  2.12.1               hd633e50_1    conda-forge
gettext                   0.21.1               h0186832_0    conda-forge
gflags                    2.2.2             hc88da5d_1004    conda-forge
git                       2.40.0          pl5321h64a6652_1    conda-forge
gitdb                     4.0.10             pyhd8ed1ab_0    conda-forge
gitpython                 3.1.30             pyhd8ed1ab_0    conda-forge
glob2                     0.7                        py_0    conda-forge
glog                      0.6.0                h6da1cb0_0    conda-forge
gmp                       6.2.1                h9f76cd9_0    conda-forge
google-auth               2.19.1                   pypi_0    pypi
greenlet                  2.0.2           py311ha891d26_1    conda-forge
html5lib                  1.1                      pypi_0    pypi
icu                       70.1                 h6b3803e_0    conda-forge
identify                  2.5.24                   pypi_0    pypi
idna                      3.4                pyhd8ed1ab_0    conda-forge
importlib-metadata        6.0.0              pyha770c72_0    conda-forge
importlib_metadata        6.0.0                hd8ed1ab_0    conda-forge
importlib_resources       5.10.2             pyhd8ed1ab_0    conda-forge
iniconfig                 2.0.0                    pypi_0    pypi
ipython                   8.13.2             pyhd1c38e8_0    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
ipywidgets                8.0.4              pyhd8ed1ab_0    conda-forge
isodate                   0.6.1              pyhd8ed1ab_0    conda-forge
jaraco-classes            3.2.3                    pypi_0    pypi
jedi                      0.18.2             pyhd8ed1ab_0    conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
joblib                    1.2.0              pyhd8ed1ab_0    conda-forge
jpeg                      9e                   h1a8c8d9_3    conda-forge
json5                     0.9.5              pyh9f0ad1d_0    conda-forge
jsonpatch                 1.32               pyhd8ed1ab_0    conda-forge
jsonpointer               2.0                        py_0    conda-forge
jsonschema                4.17.3             pyhd8ed1ab_0    conda-forge
jupyter-server-mathjax    0.2.6              pyh5bfe37b_1    conda-forge
jupyter_client            8.0.2              pyhd8ed1ab_0    conda-forge
jupyter_core              5.3.0           py311h267d04e_0    conda-forge
jupyter_events            0.5.0              pyhd8ed1ab_1    conda-forge
jupyter_server            2.2.1              pyhd8ed1ab_0    conda-forge
jupyter_server_fileid     0.6.0              pyhd8ed1ab_0    conda-forge
jupyter_server_terminals  0.4.4              pyhd8ed1ab_1    conda-forge
jupyter_server_ydoc       0.6.1              pyhd8ed1ab_0    conda-forge
jupyter_ydoc              0.2.2              pyhd8ed1ab_0    conda-forge
jupyterlab                3.6.1              pyhd8ed1ab_0    conda-forge
jupyterlab_pygments       0.2.2              pyhd8ed1ab_0    conda-forge
jupyterlab_server         2.19.0             pyhd8ed1ab_0    conda-forge
jupyterlab_widgets        3.0.5              pyhd8ed1ab_0    conda-forge
keyring                   23.13.1                  pypi_0    pypi
kiwisolver                1.4.4           py311hd6ee22a_1    conda-forge
krb5                      1.20.1               h69eda48_0    conda-forge
kubernetes                26.1.0                   pypi_0    pypi
lcms2                     2.14                 h481adae_1    conda-forge
ld64                      609                 h619f069_11    conda-forge
ld64_osx-arm64            609                 h7167370_11    conda-forge
lerc                      4.0.0                h9a09cb3_0    conda-forge
libabseil                 20230125.0      cxx17_hb7217d7_1    conda-forge
libarchive                3.6.2                h83f22c9_0    conda-forge
libarrow                  12.0.0           h5d58556_1_cpu    conda-forge
libblas                   3.9.0           16_osxarm64_openblas    conda-forge
libbrotlicommon           1.0.9                h1a8c8d9_8    conda-forge
libbrotlidec              1.0.9                h1a8c8d9_8    conda-forge
libbrotlienc              1.0.9                h1a8c8d9_8    conda-forge
libcblas                  3.9.0           16_osxarm64_openblas    conda-forge
libcrc32c                 1.1.2                hbdafb3b_0    conda-forge
libcurl                   8.0.1                heffe338_0    conda-forge
libcxx                    16.0.3               h4653b0c_0    conda-forge
libdeflate                1.17                 h1a8c8d9_0    conda-forge
libedit                   3.1.20191231         hc8eb9b7_2    conda-forge
libev                     4.33                 h642e427_1    conda-forge
libevent                  2.1.12               h5643135_0    conda-forge
libexpat                  2.5.0                hb7217d7_1    conda-forge
libffi                    3.4.2                h3422bc3_5    conda-forge
libgfortran               5.0.0           11_3_0_hd922786_27    conda-forge
libgfortran5              11.3.0              hdaf2cc0_27    conda-forge
libgoogle-cloud           2.10.0               he22f4c0_0    conda-forge
libgrpc                   1.54.1               h9dbdbd0_0    conda-forge
libiconv                  1.17                 he4db4b2_0    conda-forge
liblapack                 3.9.0           16_osxarm64_openblas    conda-forge
liblief                   0.12.3               hb7217d7_0    conda-forge
libllvm14                 14.0.6               hf6e71e7_1    conda-forge
libmamba                  1.1.0                h588180f_3    conda-forge
libmambapy                1.1.0           py311hf90d2e5_3    conda-forge
libnghttp2                1.52.0               hae82a92_0    conda-forge
libopenblas               0.3.21          openmp_hc731615_3    conda-forge
libpng                    1.6.39               h76d750c_0    conda-forge
libprotobuf               3.21.12              hb5ab8b9_0    conda-forge
libsodium                 1.0.18               h27ca646_1    conda-forge
libsolv                   0.7.23               hb5ab8b9_0    conda-forge
libsqlite                 3.40.0               h76d750c_0    conda-forge
libssh2                   1.10.0               h7a5bd25_3    conda-forge
libthrift                 0.18.1               ha061701_1    conda-forge
libtiff                   4.5.0                h5dffbdd_2    conda-forge
libutf8proc               2.8.0                h1a8c8d9_0    conda-forge
libwebp-base              1.2.4                h57fd34a_0    conda-forge
libxcb                    1.13              h9b22ae9_1004    conda-forge
libxml2                   2.10.3               h87b0503_0    conda-forge
libzlib                   1.2.13               h03a7124_4    conda-forge
license-expression        1.2                        py_0    conda-forge
llvm-openmp               15.0.7               h7cfbb63_0    conda-forge
lz4-c                     1.9.4                hb7217d7_0    conda-forge
lzo                       2.10              h642e427_1000    conda-forge
mako                      1.2.4              pyhd8ed1ab_0    conda-forge
mamba                     1.1.0           py311hb045da1_3    conda-forge
markdown-it-py            2.2.0                    pypi_0    pypi
markupsafe                2.1.2           py311he2be06e_0    conda-forge
matplotlib-base           3.7.1           py311h99a5f44_0    conda-forge
matplotlib-inline         0.1.6              pyhd8ed1ab_0    conda-forge
mdurl                     0.1.0              pyhd8ed1ab_0    conda-forge
meson                     1.1.1                    pypi_0    pypi
meson-python              0.13.1                   pypi_0    pypi
mistune                   2.0.5              pyhd8ed1ab_0    conda-forge
more-itertools            9.1.0                    pypi_0    pypi
mpi                       1.0                       mpich    conda-forge
mpi4py                    3.1.4           py311hc051084_0    conda-forge
mpich                     4.0.3              hd4b5bf3_100    conda-forge
msgpack                   1.0.5                    pypi_0    pypi
msrest                    0.6.21             pyh44b312d_0    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
mypy                      1.3.0           py311heffc1b2_0    conda-forge
mypy_extensions           1.0.0              pyha770c72_0    conda-forge
nbconvert                 7.2.9              pyhd8ed1ab_0    conda-forge
nbconvert-core            7.2.9              pyhd8ed1ab_0    conda-forge
nbconvert-pandoc          7.2.9              pyhd8ed1ab_0    conda-forge
nbdime                    3.1.1              pyhd8ed1ab_0    conda-forge
nbformat                  5.7.3              pyhd8ed1ab_0    conda-forge
ncurses                   6.3                  h07bb92c_1    conda-forge
nest-asyncio              1.5.6              pyhd8ed1ab_0    conda-forge
networkx                  3.0                pyhd8ed1ab_0    conda-forge
nodeenv                   1.8.0                    pypi_0    pypi
notebook-shim             0.2.2              pyhd8ed1ab_0    conda-forge
numpy                     1.24.3          py311hb8f3215_0    conda-forge
oauthlib                  3.2.2              pyhd8ed1ab_0    conda-forge
openjpeg                  2.5.0                hbc2ba62_2    conda-forge
openssl                   3.1.1                h53f4e23_1    conda-forge
orc                       1.8.3                hef0d403_0    conda-forge
packaging                 23.0               pyhd8ed1ab_0    conda-forge
pandas                    2.0.2           py311h9e438b8_0    conda-forge
pandoc                    2.19.2               hce30654_1    conda-forge
pandocfilters             1.5.0              pyhd8ed1ab_0    conda-forge
parquet-cpp               1.5.1                         2    conda-forge
parso                     0.8.3              pyhd8ed1ab_0    conda-forge
pastel                    0.2.1                    pypi_0    pypi
patch                     2.7.6             h27ca646_1002    conda-forge
pathspec                  0.11.1                   pypi_0    pypi
patsy                     0.5.3              pyhd8ed1ab_0    conda-forge
pcre2                     10.40                hb34f9b4_0    conda-forge
perl                      5.32.1          2_h9b22ae9_perl5    conda-forge
pexpect                   4.8.0              pyh1a96a4e_2    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    9.4.0           py311h627eb56_1    conda-forge
pip                       23.0               pyhd8ed1ab_0    conda-forge
pkginfo                   1.9.6              pyhd8ed1ab_0    conda-forge
pkgutil-resolve-name      1.3.10             pyhd8ed1ab_0    conda-forge
platformdirs              3.5.1                    pypi_0    pypi
pluggy                    1.0.0              pyhd8ed1ab_5    conda-forge
pooch                     1.6.0              pyhd8ed1ab_0    conda-forge
pre-commit                3.3.2                    pypi_0    pypi
prometheus_client         0.16.0             pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.36             pyha770c72_0    conda-forge
prompt_toolkit            3.0.36               hd8ed1ab_0    conda-forge
psutil                    5.9.5           py311he2be06e_0    conda-forge
pthread-stubs             0.4               h27ca646_1001    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
py-lief                   0.12.3          py311ha397e9f_0    conda-forge
pyarrow                   12.0.0          py311h0e52918_1_cpu    conda-forge
pyasn1                    0.5.0                    pypi_0    pypi
pyasn1-modules            0.3.0                    pypi_0    pypi
pybind11-abi              4                    hd8ed1ab_3    conda-forge
pycosat                   0.6.4           py311he2be06e_1    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pycryptodome              3.18.0          py311h7df3ba5_0    conda-forge
pycurl                    7.45.1          py311h29f4747_3    conda-forge
pydantic                  1.10.8                   pypi_0    pypi
pygithub                  1.57               pyh1a96a4e_0    conda-forge
pygments                  2.14.0             pyhd8ed1ab_0    conda-forge
pyjwt                     2.6.0              pyhd8ed1ab_0    conda-forge
pylev                     1.4.0                    pypi_0    pypi
pynacl                    1.5.0           py311he2be06e_2    conda-forge
pyopenssl                 23.2.0             pyhd8ed1ab_1    conda-forge
pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
pyproject-metadata        0.7.1                    pypi_0    pypi
pyrsistent                0.19.3          py311he2be06e_0    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
pytest                    7.3.1                    pypi_0    pypi
pytest-asyncio            0.21.0                   pypi_0    pypi
python                    3.11.3          h1456518_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-fastjsonschema     2.16.2             pyhd8ed1ab_0    conda-forge
python-json-logger        2.0.4              pyhd8ed1ab_0    conda-forge
python-libarchive-c       4.0             py311h267d04e_2    conda-forge
python-tzdata             2023.3             pyhd8ed1ab_0    conda-forge
python_abi                3.11                    3_cp311    conda-forge
pytz                      2022.7.1           pyhd8ed1ab_0    conda-forge
pyyaml                    6.0             py311he2be06e_5    conda-forge
pyzmq                     25.1.0          py311hb1af645_0    conda-forge
re2                       2023.02.02           hb7217d7_0    conda-forge
readline                  8.2                  h92ec313_1    conda-forge
readme-renderer           37.3                     pypi_0    pypi
reproc                    14.2.4               h1a8c8d9_0    conda-forge
reproc-cpp                14.2.4               hb7217d7_0    conda-forge
requests                  2.28.2             pyhd8ed1ab_0    conda-forge
requests-oauthlib         1.3.1              pyhd8ed1ab_0    conda-forge
requests-toolbelt         1.0.0                    pypi_0    pypi
rfc3986                   2.0.0                    pypi_0    pypi
rich                      13.3.1             pyhd8ed1ab_1    conda-forge
ripgrep                   13.0.0               h65448a5_2    conda-forge
rsa                       4.9                      pypi_0    pypi
ruamel.yaml               0.17.31         py311heffc1b2_0    conda-forge
ruamel.yaml.clib          0.2.7           py311he2be06e_1    conda-forge
schema                    0.7.5                    pypi_0    pypi
scipy                     1.10.1          py311h0bcca16_0    conda-forge
scrypt                    0.8.18          py311hf6ed75f_4    conda-forge
seaborn                   0.12.2               hd8ed1ab_0    conda-forge
seaborn-base              0.12.2             pyhd8ed1ab_0    conda-forge
send2trash                1.8.0              pyhd8ed1ab_0    conda-forge
setuptools                65.6.3             pyhd8ed1ab_0    conda-forge
sigtool                   0.1.3                h44b9a77_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
smmap                     3.0.5              pyh44b312d_0    conda-forge
snappy                    1.1.10               h17c5cce_0    conda-forge
sniffio                   1.3.0              pyhd8ed1ab_0    conda-forge
soupsieve                 2.3.2.post1        pyhd8ed1ab_0    conda-forge
sqlalchemy                2.0.15          py311heffc1b2_0    conda-forge
stack_data                0.6.2              pyhd8ed1ab_0    conda-forge
statsmodels               0.14.0          py311hb49d859_1    conda-forge
tabulate                  0.8.10                   pypi_0    pypi
tapi                      1100.0.11            he4954df_0    conda-forge
tbump                     6.10.0                   pypi_0    pypi
terminado                 0.17.1             pyhd1c38e8_0    conda-forge
tinycss2                  1.2.1              pyhd8ed1ab_0    conda-forge
tk                        8.6.12               he1e0b03_0    conda-forge
toml                      0.10.2             pyhd8ed1ab_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
tomlkit                   0.11.8                   pypi_0    pypi
toolz                     0.12.0             pyhd8ed1ab_0    conda-forge
tornado                   6.3.2           py311heffc1b2_0    conda-forge
tqdm                      4.64.1             pyhd8ed1ab_0    conda-forge
traitlets                 5.9.0              pyhd8ed1ab_0    conda-forge
twine                     4.0.2                    pypi_0    pypi
typing-extensions         4.4.0                hd8ed1ab_0    conda-forge
typing_extensions         4.4.0              pyha770c72_0    conda-forge
tzdata                    2022g                h191b570_0    conda-forge
unicodedata2              15.0.0          py311he2be06e_0    conda-forge
unidecode                 1.3.6                    pypi_0    pypi
urllib3                   1.26.14            pyhd8ed1ab_0    conda-forge
virtualenv                20.23.0                  pypi_0    pypi
vsts-python-api           0.1.25             pyhd8ed1ab_1    conda-forge
watchgod                  0.8.2              pyhd8ed1ab_0    conda-forge
wcwidth                   0.2.6              pyhd8ed1ab_0    conda-forge
webencodings              0.5.1                      py_1    conda-forge
websocket-client          1.5.1              pyhd8ed1ab_0    conda-forge
wheel                     0.38.4             pyhd8ed1ab_0    conda-forge
widgetsnbextension        4.0.5              pyhd8ed1ab_0    conda-forge
wrapt                     1.15.0          py311he2be06e_0    conda-forge
xorg-libxau               1.0.9                h27ca646_0    conda-forge
xorg-libxdmcp             1.1.3                h27ca646_0    conda-forge
xz                        5.2.6                h57fd34a_0    conda-forge
y-py                      0.5.9           py311heb423af_0    conda-forge
yaml                      0.2.5                h3422bc3_2    conda-forge
yaml-cpp                  0.7.0                hb7217d7_2    conda-forge
ypy-websocket             0.8.2              pyhd8ed1ab_0    conda-forge
zeromq                    4.3.4                hbdafb3b_1    conda-forge
zipp                      3.13.0             pyhd8ed1ab_0    conda-forge
zlib                      1.2.13               h03a7124_4    conda-forge
zstandard                 0.19.0          py311hdcbfb07_1    conda-forge
zstd                      1.5.2                hf913c23_6    conda-forge

Additional Context

A simple env that produces the issue is:

channels:
  - conda-forge
dependencies:
  - tbb

which includes libhwloc cuda variant and cudatoolkit by default on linux and Windows.
@maresb
Copy link
Contributor

maresb commented Jun 2, 2023

I don't really use cuda much, so hopefully @mariusvniekerk can chime in. It seems reasonable to me, although a fairly significant breaking change.

It anyways looks like we're using CUDA 11.4 virtual package in our fake repodata, even though the latest is 12.1. Perhaps there are multiple todos here on the virtual package front?

@minrk
Copy link
Contributor Author

minrk commented Jun 8, 2023

I think probably the biggest counter argument is that most cuda things explicitly depend on cuda, and some of those environments will no longer be solvable without specifying a virtual package spec anymore, and it's uncommon for things that don't depend on cuda that have a cuda variant. I get the impression that this is becoming less true, though. I just happen to be using one of those packages (torch-cpu -> mkl -> tbb -> hwloc -> cudatoolkit).

@maresb
Copy link
Contributor

maresb commented Jun 8, 2023

Would it make sense to introduce --with-cuda and --without-cuda flags? And then if something CUDA is installed and these flags aren't specified, then we emit a warning asking the user to be explicit?

@minrk
Copy link
Contributor Author

minrk commented Jun 8, 2023

That would mean keeping cuda in the default virtual packages (otherwise it would fail to solve and we wouldn't get to a warning), then checking for e.g. cudatoolkit in the result and warning if cuda was left unspecified (no virtual packages, no --with[out]-cuda)? That seems like reasonable behavior. A bit more complex to implement, but not too bad.

@maresb
Copy link
Contributor

maresb commented Jun 8, 2023

This seems to be quite doable to me, and it sounds like this would have prevented the need for you to debug the image size.

I'd be happy to accept a PR. I'm a bit time-constrained, but I might be able to get to this several months from now.

@minrk
Copy link
Contributor Author

minrk commented Jun 8, 2023

I'll have a look if I get a chance.

This was referenced Jun 8, 2023
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

warn if cuda packages are installed implicitly minrk/conda-lock
2 participants
@minrk @maresb