Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Channel priority not maximized over competing goals #532

Closed
mbargull opened this issue Oct 13, 2020 · 7 comments
Closed

Channel priority not maximized over competing goals #532

mbargull opened this issue Oct 13, 2020 · 7 comments
Labels
where::libsolv

Comments

@mbargull
Copy link
Contributor

mamba seemingly prioritizes the package count minimization or package version maximization over channel priority.
In the case below it is expected (at least with conda's semantics) that conda-forge::libxcb gets chosen, but mamba chooses defaults::libxcb.
(For this particular case, the source channel is important because conda-forge::libxcb depends on xorg-* packages and as such is usable in base-minimum containers, whereas defaults::libxcb uses CDT packages and thus is not usable in self-contained environments.)

$ mamba --version
mamba 0.5.3
conda 4.8.5
$ diff \
>   <( mamba create -dqnx -c conda-forge graphviz=2.42.3=h6939c30_1 2>/dev/null ) \
>   <( mamba create -dqnx -c conda-forge graphviz=2.42.3=h6939c30_1 libxcb 2>/dev/null )
$ diff \
>   <( mamba create -dqnx -c conda-forge graphviz=2.42.3=h6939c30_1 2>/dev/null ) \
>   <( mamba create -dqnx -c conda-forge graphviz=2.42.3=h6939c30_1 conda-forge::libxcb 2>/dev/null )
47c47
<   libxcb                          1.14  h7b6447c_0          pkgs/main/linux-64       505 KB
---
>   libxcb                          1.13  h14c3975_1002       conda-forge/linux-64     396 KB
55a56
>   pthread-stubs                    0.4  h14c3975_1001       conda-forge/linux-64       5 KB
67a69,70
>   xorg-libxau                    1.0.9  h14c3975_0          conda-forge/linux-64      13 KB
>   xorg-libxdmcp                  1.1.3  h516909a_0          conda-forge/linux-64      18 KB
81c84
<   Install: 72 packages
---
>   Install: 75 packages

Below are the complete package lists from mamba create and conda create, resp., for comparison:

$ mamba create -dqnx -c conda-forge graphviz=2.42.3=h6939c30_1 2>/dev/null
 
  Package                      Version  Build               Channel                    Size
─────────────────────────────────────────────────────────────────────────────────────────────
  Install:
─────────────────────────────────────────────────────────────────────────────────────────────

  _libgcc_mutex                    0.1  conda_forge         conda-forge/linux-64     Cached
  _openmp_mutex                    4.5  1_gnu               conda-forge/linux-64     Cached
  atk                           2.36.0  2                   conda-forge/linux-64      12 KB
  atk-1.0                       2.36.0  h516909a_2          conda-forge/linux-64     536 KB
  ca-certificates            2020.7.22  0                   pkgs/main/linux-64       125 KB
  cairo                         1.16.0  h3fc0475_1005       conda-forge/linux-64       1 MB
  certifi                    2020.6.20  py39h079e4ff_2      conda-forge/linux-64     151 KB
  expat                         2.2.10  he6710b0_2          pkgs/main/linux-64       153 KB
  font-ttf-dejavu-sans-mono       2.37  hab24e00_0          conda-forge/noarch       388 KB
  font-ttf-inconsolata           2.001  hab24e00_0          conda-forge/noarch        97 KB
  font-ttf-source-code-pro       2.030  hab24e00_0          conda-forge/noarch       927 KB
  font-ttf-ubuntu                 0.83  hab24e00_0          conda-forge/noarch         2 MB
  fontconfig                    2.13.1  h1056068_1002       conda-forge/linux-64     365 KB
  fonts-conda-ecosystem              1  0                   conda-forge/noarch         4 KB
  fonts-conda-forge                  1  0                   conda-forge/noarch         4 KB
  freetype                      2.10.3  he06d7ca_0          conda-forge/linux-64     920 KB
  fribidi                       1.0.10  h516909a_0          conda-forge/linux-64     113 KB
  gdk-pixbuf                    2.38.2  h3f25603_4          conda-forge/linux-64     653 KB
  gettext                     0.19.8.1  hc5be6a0_1002       conda-forge/linux-64     Cached
  giflib                         5.2.1  h516909a_2          conda-forge/linux-64      80 KB
  glib                          2.66.1  h680cd38_0          conda-forge/linux-64       3 MB
  gobject-introspection         1.66.1  py39hf8e0571_1      conda-forge/linux-64       1 MB
  graphite2                     1.3.14  h23475e2_0          pkgs/main/linux-64        99 KB
  graphviz                      2.42.3  h6939c30_1          conda-forge/linux-64       7 MB
  gtk2                         2.24.32  h194ddfc_3          conda-forge/linux-64       7 MB
  gts                            0.7.6  h08bb679_0          conda-forge/linux-64     400 KB
  harfbuzz                       2.7.2  hee91db6_0          conda-forge/linux-64       2 MB
  icu                             67.1  he1b5a44_0          conda-forge/linux-64     Cached
  jpeg                              9d  h516909a_0          conda-forge/linux-64     266 KB
  ld_impl_linux-64                2.35  h769bd43_9          conda-forge/linux-64     617 KB
  libffi                         3.2.1  he1b5a44_1007       conda-forge/linux-64     Cached
  libgcc-ng                      9.3.0  h5dbcf3e_17         conda-forge/linux-64       8 MB
  libgomp                        9.3.0  h5dbcf3e_17         conda-forge/linux-64     378 KB
  libiconv                        1.16  h516909a_0          conda-forge/linux-64     Cached
  libpng                        1.6.37  hed695b0_2          conda-forge/linux-64     359 KB
  libstdcxx-ng                   9.3.0  h2ae2ef3_17         conda-forge/linux-64       4 MB
  libtiff                        4.1.0  hc7e4089_6          conda-forge/linux-64     668 KB
  libtool                        2.4.6  h516909a_1005       conda-forge/linux-64     498 KB
  libuuid                       2.32.1  h14c3975_1000       conda-forge/linux-64      26 KB
  libwebp                        1.1.0  h56121f0_4          conda-forge/linux-64      85 KB
  libwebp-base                   1.1.0  h516909a_3          conda-forge/linux-64     845 KB
  libxcb                          1.14  h7b6447c_0          pkgs/main/linux-64       505 KB
  libxml2                       2.9.10  h68273f3_2          conda-forge/linux-64       1 MB
  lz4-c                          1.9.2  he1b5a44_3          conda-forge/linux-64     203 KB
  ncurses                          6.2  he1b5a44_2          conda-forge/linux-64     991 KB
  openssl                       1.1.1h  h516909a_0          conda-forge/linux-64     Cached
  pango                         1.42.4  h7062337_4          conda-forge/linux-64     521 KB
  pcre                            8.44  he1b5a44_0          conda-forge/linux-64     Cached
  pip                           20.2.3  py_0                conda-forge/noarch         1 MB
  pixman                        0.38.0  h516909a_1003       conda-forge/linux-64     594 KB
  python                         3.9.0  h852b56e_3_cpython  conda-forge/linux-64      24 MB
  python_abi                       3.9  1_cp39              conda-forge/linux-64       4 KB
  readline                         8.0  he28a2e2_2          conda-forge/linux-64     Cached
  setuptools                    49.6.0  py39h079e4ff_2      conda-forge/linux-64     953 KB
  sqlite                        3.33.0  h4cf870e_1          conda-forge/linux-64       1 MB
  tk                            8.6.10  hed695b0_1          conda-forge/linux-64       3 MB
  tzdata                         2020b  h516909a_0          conda-forge/noarch       120 KB
  wheel                         0.35.1  pyh9f0ad1d_0        conda-forge/noarch        29 KB
  xorg-kbproto                   1.0.7  h14c3975_1002       conda-forge/linux-64      26 KB
  xorg-libice                   1.0.10  h516909a_0          conda-forge/linux-64      57 KB
  xorg-libsm                     1.2.3  h84519dc_1000       conda-forge/linux-64      25 KB
  xorg-libx11                   1.6.12  h516909a_0          conda-forge/linux-64     917 KB
  xorg-libxext                   1.3.4  h516909a_0          conda-forge/linux-64      51 KB
  xorg-libxpm                   3.5.13  h516909a_0          conda-forge/linux-64      63 KB
  xorg-libxrender               0.9.10  h516909a_1002       conda-forge/linux-64      31 KB
  xorg-libxt                     1.1.5  h516909a_1003       conda-forge/linux-64     367 KB
  xorg-renderproto              0.11.1  h14c3975_1002       conda-forge/linux-64       8 KB
  xorg-xextproto                 7.3.0  h14c3975_1002       conda-forge/linux-64      27 KB
  xorg-xproto                   7.0.31  h14c3975_1007       conda-forge/linux-64      72 KB
  xz                             5.2.5  h516909a_1          conda-forge/linux-64     Cached
  zlib                          1.2.11  h516909a_1009       conda-forge/linux-64     106 KB
  zstd                           1.4.5  h6597ccf_2          conda-forge/linux-64     Cached

  Summary:

  Install: 72 packages

  Total download: 81 MB

─────────────────────────────────────────────────────────────────────────────────────────────

$ conda create -dqnx -c conda-forge graphviz=2.42.3=h6939c30_1 2>/dev/null
 
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done

## Package Plan ##

  environment location: /opt/conda/envs/x

  added / updated specs:
    - graphviz==2.42.3=h6939c30_1


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    atk-2.36.0                 |                2          12 KB  conda-forge
    atk-1.0-2.36.0             |       h516909a_2         536 KB  conda-forge
    cairo-1.16.0               |    h3fc0475_1005         1.5 MB  conda-forge
    certifi-2020.6.20          |   py39h079e4ff_2         151 KB  conda-forge
    font-ttf-dejavu-sans-mono-2.37|       hab24e00_0         388 KB  conda-forge
    font-ttf-inconsolata-2.001 |       hab24e00_0          97 KB  conda-forge
    font-ttf-source-code-pro-2.030|       hab24e00_0         927 KB  conda-forge
    font-ttf-ubuntu-0.83       |       hab24e00_0         1.9 MB  conda-forge
    fontconfig-2.13.1          |    h1056068_1002         365 KB  conda-forge
    fonts-conda-ecosystem-1    |                0           4 KB  conda-forge
    fonts-conda-forge-1        |                0           4 KB  conda-forge
    freetype-2.10.3            |       he06d7ca_0         920 KB  conda-forge
    fribidi-1.0.10             |       h516909a_0         113 KB  conda-forge
    gdk-pixbuf-2.38.2          |       h3f25603_4         653 KB  conda-forge
    giflib-5.2.1               |       h516909a_2          80 KB  conda-forge
    glib-2.66.1                |       h680cd38_0         3.5 MB  conda-forge
    gobject-introspection-1.66.1|   py39hf8e0571_1         1.4 MB  conda-forge
    graphite2-1.3.13           |    he1b5a44_1001         102 KB  conda-forge
    graphviz-2.42.3            |       h6939c30_1         7.2 MB  conda-forge
    gtk2-2.24.32               |       h194ddfc_3         7.3 MB  conda-forge
    gts-0.7.6                  |       h08bb679_0         400 KB  conda-forge
    harfbuzz-2.7.2             |       hee91db6_0         1.8 MB  conda-forge
    jpeg-9d                    |       h516909a_0         266 KB  conda-forge
    ld_impl_linux-64-2.35      |       h769bd43_9         617 KB  conda-forge
    libgcc-ng-9.3.0            |      h5dbcf3e_17         7.8 MB  conda-forge
    libgomp-9.3.0              |      h5dbcf3e_17         378 KB  conda-forge
    libpng-1.6.37              |       hed695b0_2         359 KB  conda-forge
    libstdcxx-ng-9.3.0         |      h2ae2ef3_17         4.0 MB  conda-forge
    libtiff-4.1.0              |       hc7e4089_6         668 KB  conda-forge
    libtool-2.4.6              |    h516909a_1005         498 KB  conda-forge
    libuuid-2.32.1             |    h14c3975_1000          26 KB  conda-forge
    libwebp-1.1.0              |       h56121f0_4          85 KB  conda-forge
    libwebp-base-1.1.0         |       h516909a_3         845 KB  conda-forge
    libxcb-1.13                |    h14c3975_1002         396 KB  conda-forge
    libxml2-2.9.10             |       h68273f3_2         1.3 MB  conda-forge
    lz4-c-1.9.2                |       he1b5a44_3         203 KB  conda-forge
    ncurses-6.2                |       he1b5a44_2         991 KB  conda-forge
    pango-1.42.4               |       h7062337_4         521 KB  conda-forge
    pip-20.2.3                 |             py_0         1.1 MB  conda-forge
    pixman-0.38.0              |    h516909a_1003         594 KB  conda-forge
    pthread-stubs-0.4          |    h14c3975_1001           5 KB  conda-forge
    python-3.9.0               |h852b56e_3_cpython        24.0 MB  conda-forge
    python_abi-3.9             |           1_cp39           4 KB  conda-forge
    setuptools-49.6.0          |   py39h079e4ff_2         953 KB  conda-forge
    sqlite-3.33.0              |       h4cf870e_1         1.4 MB  conda-forge
    tk-8.6.10                  |       hed695b0_1         3.2 MB  conda-forge
    tzdata-2020b               |       h516909a_0         120 KB  conda-forge
    wheel-0.35.1               |     pyh9f0ad1d_0          29 KB  conda-forge
    xorg-kbproto-1.0.7         |    h14c3975_1002          26 KB  conda-forge
    xorg-libice-1.0.10         |       h516909a_0          57 KB  conda-forge
    xorg-libsm-1.2.3           |    h84519dc_1000          25 KB  conda-forge
    xorg-libx11-1.6.12         |       h516909a_0         917 KB  conda-forge
    xorg-libxau-1.0.9          |       h14c3975_0          13 KB  conda-forge
    xorg-libxdmcp-1.1.3        |       h516909a_0          18 KB  conda-forge
    xorg-libxext-1.3.4         |       h516909a_0          51 KB  conda-forge
    xorg-libxpm-3.5.13         |       h516909a_0          63 KB  conda-forge
    xorg-libxrender-0.9.10     |    h516909a_1002          31 KB  conda-forge
    xorg-libxt-1.1.5           |    h516909a_1003         367 KB  conda-forge
    xorg-renderproto-0.11.1    |    h14c3975_1002           8 KB  conda-forge
    xorg-xextproto-7.3.0       |    h14c3975_1002          27 KB  conda-forge
    xorg-xproto-7.0.31         |    h14c3975_1007          72 KB  conda-forge
    zlib-1.2.11                |    h516909a_1009         106 KB  conda-forge
    ------------------------------------------------------------
                                           Total:        81.1 MB

The following NEW packages will be INSTALLED:

  _libgcc_mutex      conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge
  _openmp_mutex      conda-forge/linux-64::_openmp_mutex-4.5-1_gnu
  atk                conda-forge/linux-64::atk-2.36.0-2
  atk-1.0            conda-forge/linux-64::atk-1.0-2.36.0-h516909a_2
  ca-certificates    conda-forge/linux-64::ca-certificates-2020.6.20-hecda079_0
  cairo              conda-forge/linux-64::cairo-1.16.0-h3fc0475_1005
  certifi            conda-forge/linux-64::certifi-2020.6.20-py39h079e4ff_2
  expat              conda-forge/linux-64::expat-2.2.9-he1b5a44_2
  font-ttf-dejavu-s~ conda-forge/noarch::font-ttf-dejavu-sans-mono-2.37-hab24e00_0
  font-ttf-inconsol~ conda-forge/noarch::font-ttf-inconsolata-2.001-hab24e00_0
  font-ttf-source-c~ conda-forge/noarch::font-ttf-source-code-pro-2.030-hab24e00_0
  font-ttf-ubuntu    conda-forge/noarch::font-ttf-ubuntu-0.83-hab24e00_0
  fontconfig         conda-forge/linux-64::fontconfig-2.13.1-h1056068_1002
  fonts-conda-ecosy~ conda-forge/noarch::fonts-conda-ecosystem-1-0
  fonts-conda-forge  conda-forge/noarch::fonts-conda-forge-1-0
  freetype           conda-forge/linux-64::freetype-2.10.3-he06d7ca_0
  fribidi            conda-forge/linux-64::fribidi-1.0.10-h516909a_0
  gdk-pixbuf         conda-forge/linux-64::gdk-pixbuf-2.38.2-h3f25603_4
  gettext            conda-forge/linux-64::gettext-0.19.8.1-hc5be6a0_1002
  giflib             conda-forge/linux-64::giflib-5.2.1-h516909a_2
  glib               conda-forge/linux-64::glib-2.66.1-h680cd38_0
  gobject-introspec~ conda-forge/linux-64::gobject-introspection-1.66.1-py39hf8e0571_1
  graphite2          conda-forge/linux-64::graphite2-1.3.13-he1b5a44_1001
  graphviz           conda-forge/linux-64::graphviz-2.42.3-h6939c30_1
  gtk2               conda-forge/linux-64::gtk2-2.24.32-h194ddfc_3
  gts                conda-forge/linux-64::gts-0.7.6-h08bb679_0
  harfbuzz           conda-forge/linux-64::harfbuzz-2.7.2-hee91db6_0
  icu                conda-forge/linux-64::icu-67.1-he1b5a44_0
  jpeg               conda-forge/linux-64::jpeg-9d-h516909a_0
  ld_impl_linux-64   conda-forge/linux-64::ld_impl_linux-64-2.35-h769bd43_9
  libffi             conda-forge/linux-64::libffi-3.2.1-he1b5a44_1007
  libgcc-ng          conda-forge/linux-64::libgcc-ng-9.3.0-h5dbcf3e_17
  libgomp            conda-forge/linux-64::libgomp-9.3.0-h5dbcf3e_17
  libiconv           conda-forge/linux-64::libiconv-1.16-h516909a_0
  libpng             conda-forge/linux-64::libpng-1.6.37-hed695b0_2
  libstdcxx-ng       conda-forge/linux-64::libstdcxx-ng-9.3.0-h2ae2ef3_17
  libtiff            conda-forge/linux-64::libtiff-4.1.0-hc7e4089_6
  libtool            conda-forge/linux-64::libtool-2.4.6-h516909a_1005
  libuuid            conda-forge/linux-64::libuuid-2.32.1-h14c3975_1000
  libwebp            conda-forge/linux-64::libwebp-1.1.0-h56121f0_4
  libwebp-base       conda-forge/linux-64::libwebp-base-1.1.0-h516909a_3
  libxcb             conda-forge/linux-64::libxcb-1.13-h14c3975_1002
  libxml2            conda-forge/linux-64::libxml2-2.9.10-h68273f3_2
  lz4-c              conda-forge/linux-64::lz4-c-1.9.2-he1b5a44_3
  ncurses            conda-forge/linux-64::ncurses-6.2-he1b5a44_2
  openssl            conda-forge/linux-64::openssl-1.1.1h-h516909a_0
  pango              conda-forge/linux-64::pango-1.42.4-h7062337_4
  pcre               conda-forge/linux-64::pcre-8.44-he1b5a44_0
  pip                conda-forge/noarch::pip-20.2.3-py_0
  pixman             conda-forge/linux-64::pixman-0.38.0-h516909a_1003
  pthread-stubs      conda-forge/linux-64::pthread-stubs-0.4-h14c3975_1001
  python             conda-forge/linux-64::python-3.9.0-h852b56e_3_cpython
  python_abi         conda-forge/linux-64::python_abi-3.9-1_cp39
  readline           conda-forge/linux-64::readline-8.0-he28a2e2_2
  setuptools         conda-forge/linux-64::setuptools-49.6.0-py39h079e4ff_2
  sqlite             conda-forge/linux-64::sqlite-3.33.0-h4cf870e_1
  tk                 conda-forge/linux-64::tk-8.6.10-hed695b0_1
  tzdata             conda-forge/noarch::tzdata-2020b-h516909a_0
  wheel              conda-forge/noarch::wheel-0.35.1-pyh9f0ad1d_0
  xorg-kbproto       conda-forge/linux-64::xorg-kbproto-1.0.7-h14c3975_1002
  xorg-libice        conda-forge/linux-64::xorg-libice-1.0.10-h516909a_0
  xorg-libsm         conda-forge/linux-64::xorg-libsm-1.2.3-h84519dc_1000
  xorg-libx11        conda-forge/linux-64::xorg-libx11-1.6.12-h516909a_0
  xorg-libxau        conda-forge/linux-64::xorg-libxau-1.0.9-h14c3975_0
  xorg-libxdmcp      conda-forge/linux-64::xorg-libxdmcp-1.1.3-h516909a_0
  xorg-libxext       conda-forge/linux-64::xorg-libxext-1.3.4-h516909a_0
  xorg-libxpm        conda-forge/linux-64::xorg-libxpm-3.5.13-h516909a_0
  xorg-libxrender    conda-forge/linux-64::xorg-libxrender-0.9.10-h516909a_1002
  xorg-libxt         conda-forge/linux-64::xorg-libxt-1.1.5-h516909a_1003
  xorg-renderproto   conda-forge/linux-64::xorg-renderproto-0.11.1-h14c3975_1002
  xorg-xextproto     conda-forge/linux-64::xorg-xextproto-7.3.0-h14c3975_1002
  xorg-xproto        conda-forge/linux-64::xorg-xproto-7.0.31-h14c3975_1007
  xz                 conda-forge/linux-64::xz-5.2.5-h516909a_1
  zlib               conda-forge/linux-64::zlib-1.2.11-h516909a_1009
  zstd               conda-forge/linux-64::zstd-1.4.5-h6597ccf_2

(cc @FelixMoelder who encountered this in the snakemake/snakemake:v5.26.1 container image.)
@wolfv
Copy link
Member

wolfv commented Oct 13, 2020

even with strict-channel-priority? That is (afaik) recommended for the conda-forge use-case.

@mbargull
Copy link
Contributor Author

If mamba functions the same as conda with channel_priority: strict, then this can't happen with strict, of course.

So, are you saying mamba does not properly support the channel_priority: flexible use case?
If yes, that restriction should be added to the docs.
(cc @johanneskoester for viz)

@wolfv
Copy link
Member

wolfv commented Jun 30, 2021

@mbargull with a flexible channel priority, mamba will always choose the higher version, independent of the channel it came from.

Although I just re-checked the docs on conda and it seems that indeed, I misinterpreted them before.
I think our current "strict" behavior is closer to the "flexible" behavior of conda, and our flexible behavior is closer to the no channel priority / channel_priority: false behavior of conda.

https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-channels.html#managing-channels

There is a similar issue for our (too lax) strict channel priority: #559

@wolfv wolfv mentioned this issue Jun 30, 2021
Open
17 tasks
@wolfv
Copy link
Member

wolfv commented Jul 8, 2021

@mbargull we've changed the default now to better match conda's behavior:

  • strict is strict behavior (really completely ignoring packages from lower-prio channels)
  • with flexible, packages from lower prio can still be selected, but only directly and if there is no alternative
  • with disabled only the version number is taken into account

@wolfv wolfv closed this as completed Jul 8, 2021
@mbargull
Copy link
Contributor Author

mbargull commented Jul 9, 2021

Sounds good, thanks Wolf, very much appreciated!

  • with flexible, packages from lower prio can still be selected, but only directly and if there is no alternative

What does "only directly" mean in this context?

@wolfv
Copy link
Member

wolfv commented Jul 9, 2021

let's say you have

channel A (prio 10)

  • numpy 2.0
  • numpy 2.1
  • pandas 1.0 -> depends on numpy 1.0.*

channel B (prio 0)

  • numpy 1.0.1

Now if you have flexible prio you can do install numpy=1 and it will pull numpy from channel B

Similarly if you install pandas it will pull numpy from channel B since it depends can only be fulfilled by channel B.

@mbargull
Copy link
Contributor Author

mbargull commented Jul 9, 2021

Perfect, thanks again!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
where::libsolv
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@wolfv @mbargull