uniform sat solving/resolution accross features to form one consistent environment

[geoHeil]

As a developer I want to define shared base dependencies in one feature, and reference it in one or more other environments; I would expect that solving for the dependencies is performed in a holistic/combined way so that even combining feature a + b in environment foo works.

However, I realized - see the example, that this assumption is not valid.

option 1: 2 packages are combined into one environment via features

This fails - why?

[dependencies]
python = "3.11"
pip = ">=24.2"

[environments]
step-one = {features = ["fooa", ], solve-group = "xx"}
step-two = {features = [ "foob",   "fooa", ], solve-group = "xx"}

[feature.fooa.pypi-dependencies]
dbt-core = ">=1.8.5,<1.9"

[feature.foob.dependencies]
jupyterlab = "~=4.2.4"


[project]
name = "foo"
version = "1.0.0"
description = "bar"
authors = [
  "baz"
]
platforms = ["linux-64","osx-arm64"]
channels = ["conda-forge"]

error log:

 failed to resolve pypi dependencies
  ╰─▶ Because dbt-semantic-interfaces==0.5.1 depends on importlib-metadata>=6.0,<7 and importlib-metadata==8.2.0, we can
      conclude that dbt-semantic-interfaces==0.5.1 cannot be used.
      And because only the following versions of dbt-semantic-interfaces are available:
          dbt-semantic-interfaces<=0.5.1
          dbt-semantic-interfaces>=0.6
      and dbt-core==1.8.5 depends on dbt-semantic-interfaces>=0.5.1,<0.6, we can conclude that dbt-core==1.8.5 cannot be used.
      And because only the following versions of dbt-core are available:
          dbt-core<=1.8.5
          dbt-core>=1.9
      and you require dbt-core>=1.8.5, we can conclude that the requirements are unsatisfiable.

option 2 - minimal change same environment

still both packages are combined in one environment

This works!

The following (observe only the minimal change moving the dependency up to the same feature -- works just fine

[dependencies]
python = "3.11"
pip = ">=24.2"

[environments]
step-one = {features = ["fooa", ], solve-group = "xx"}
step-two = {features = [ "foob",   "fooa", ], solve-group = "xx"}

[feature.fooa.pypi-dependencies]
dbt-core = ">=1.8.5,<1.9"
jupyterlab = "~=4.2.4"

[feature.foob.dependencies]



[project]
name = "foo"
version = "1.0.0"
description = "bar"
authors = [
  "baz"
]
platforms = ["linux-64","osx-arm64"]
channels = ["conda-forge"]

[geoHeil]

Observation - even moving one package to the default dependencies - does not fix the issue. How can this be? The default one is included in solving the feature/environment by default?

And why does it work if they sit both in the same environment

[dependencies]
python = "3.11"
pip = ">=24.2"

[pypi-dependencies]
dbt-core = ">=1.8.5,<1.9"

[environments]
step-one = {features = ["fooa", ], solve-group = "xx"}
step-two = {features = [ "fooa", "foob",    ], solve-group = "xy"}

[feature.fooa.pypi-dependencies]


[feature.foob.dependencies]
jupyterlab = "~=4.2.4"


[project]
name = "foo"
version = "1.0.0"
description = "bar"
authors = [
  "baz"
]
platforms = ["linux-64","osx-arm64"]
channels = ["conda-forge"]

[ruben-arts]

The problem lays with the [dependencies] vs [pypi-dependencies]. These are two SAT problems. I've tried to put it in a little diagram to explain the steps. Note that we use two different resolvers to solve the SAT problem, rattler for conda, and uv for pypi.

For your example, you had jupyterlab as a conda dependency which gave the PyPI solve little options for the dbt-core dependencies to solve with.

Putting both dependencies in pypi-dependencies solve the issue for you as it now didn't get any given pypi package from the conda packages thus could search the whole set of options on https://pypi.org/

I hope this explains it.

[geoHeil]

Yes - as a workaround. But This is rather inconvenient and makes it hard to mix packages from pip/conda where I try to use as much conda as possible - but sometimes have to resort to pip. In fact, for some combinations (rapids, dbt) resolution will be outright impossible as one must be conda - and the other one pip.

Do you think there is any chance that a more unified SAT resolution procedure will be handled in the future?

For example here:

failed to solve the conda requirements of 'step-one' 'osx-arm64'
  ╰─▶ Cannot solve the request because of: dbt-core >=1.8.5,<1.9 cannot be installed because there are no viable options:
      └─ dbt-core 1.8.5 would require
         └─ agate >=1.7.0,<1.10, which cannot be installed because there are no viable options:
            └─ agate 1.8.0 | 1.8.0 | 1.9.0 | 1.9.0 | 1.9.1 | 1.9.1 would require
               └─ pyicu *, for which no candidates were found.

for

[dependencies]
python = "3.11"
pip = ">=24.2"

[pypi-dependencies]


[environments]
step-one = {features = ["fooa", ], solve-group = "xx"}
step-two = {features = [ "fooa", "foob",    ], solve-group = "xy"}

[feature.fooa.pypi-dependencies]
#dbt-core = ">=1.8.5,<1.9"
[feature.fooa.dependencies]
dbt-core = ">=1.8.5,<1.9"

[feature.foob.dependencies]
jupyterlab = "~=4.2.4"


[project]
name = "foo"
version = "1.0.0"
description = "bar"
authors = [
  "baz"
]
platforms = ["linux-64","osx-arm64"]
channels = ["conda-forge"]

Indeed:

[dependencies]
python = "3.11"
pip = ">=24.2"

[pypi-dependencies]


[environments]
step-one = {features = ["fooa", ], solve-group = "xx"}
step-two = {features = [ "fooa", "foob",    ], solve-group = "xy"}

[feature.fooa.pypi-dependencies]
dbt-core = ">=1.8.5,<1.9"
[feature.fooa.dependencies]
#dbt-core = ">=1.8.5,<1.9"

[feature.foob.dependencies]
[feature.foob.pypi-dependencies]
jupyterlab = "~=4.2.4"


[project]
name = "foo"
version = "1.0.0"
description = "bar"
authors = [
  "baz"
]
platforms = ["linux-64","osx-arm64"]
channels = ["conda-forge"]

resolves - but identifying which packages to move can be time consuming - is there any chance there is some tooling in pixi to quickly identify such / all resolution problems?

[geoHeil]

What I mean is - the tree command fails - as it does not resolve so I cannot use it for debugging

[ruben-arts]

This will not be viable in the near future. As the two problems have different solvers. Getting uv and rattler on the same solver is hard and a political. But it does fit in the vision we have for pixi so it's not a "never".

[ruben-arts]

You can help yourself and the community by putting the packages you require from pypi on conda-forge which will avoid this problem.