pixi global: environment variables (CONDA_PREFIX,PATH,etc.) in installed scripts break certain packages

[183amir]

Checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pixi, using pixi --version.

Reproducible example

pixi global install uv
conda create -n some_env python
conda activate some_env
uv pip install numpy

fails with:

error: failed to canonicalize path `~/.pixi/envs/uv/bin/python`
  Caused by: No such file or directory (os error 2)

Issue description

I have installed uv with pixi using the command:

pixi global install uv

which installs uv in a conda environment in ~/.pixi/envs/uv where the uv binary is installed in ~/.pixi/envs/uv/bin/uv and a bash script named ~/.pixi/bin/uv is created with the following content:

#!/bin/sh
export PATH="~/.pixi/envs/uv/bin:${PATH}"
export CONDA_PREFIX="~/.pixi/envs/uv"
"~/.pixi/envs/uv/bin/uv" "$@"

but the exported CONDA_PREFIX variable breaks uv since it is meant to work on currently activated conda environment.

Expected behavior

I expect to be able to install and use uv with pixi global install. However due to the nature of uv and how pixi creates global installations, uv breaks. I am not sure this is a bug in uv or pixi but would be nice to be able to install uv with pixi.

Note that I am not trying to use uv to install packages in a pixi environment. I know that pixi integrates uv internally. I am trying to use uv to pip install some packages in a conda environment.

[tdejager]

Yeah I saw this as well. Not sure what's causing this, we should have a closer look!

[183amir]

The cause is the exported CONDA_PREFIX variable in the ~/.pixi/bin/uv bash script. uv uses this CONDA_PREFIX variable to determine the currently activated conda environment. I have removed that line and now uv works but would nice if this was fixed here.

[tdejager]

I'm not sure about these things, but shouldn't python be a runtime dependency for uv? As you need this for using uv, and I would suppose adding that would install it into the global prefix? https://github.com/conda-forge/uv-feedstock

Pinging @wolfv @ruben-arts who should know more about these things :)

Edit: and @bollwyvl because he is a maintainer as well

[wolfv]

Yeah, @183amir you correctly identified the problem. I am unsure how we should fix this.

We have some other problems with global install so I think over time we want to introduce more configuration to the "global" installs - and one of the config options could be about the CONDA_PREFIX.

Hypothetically it might also be nice to have some shared configuration for all pixi users that covers some cases like this (either as annotations in the packages or as some external file).

[bollwyvl]

The initial stab at uv had python as a dependency, but on review in staged-recipes, it was noted that it was shipping one file to $PREFIX/bin, and didn't link against python. From a conda-forge perspective, this meant we could get away with ~5 {arch} * {platform} builds instead of the * {py} matrix factor for ~25 builds.

We can adopt something like the Arch pattern, where they have a noarch python-uv which depends on uv (the binary), potentially even in the same build (e.g. only build on linux x 64 x py3.12), but could run with any supported conda-forge python.

[zen-xu]

Is there any reason why symbolic links are not used to export global installed bins?

And I tried using a symbolic link to uv and it seems to resolve this issue.

[183amir]

I'm not sure about these things, but shouldn't python be a runtime dependency for uv? As you need this for using uv, and I would suppose adding that would install it into the global prefix? https://github.com/conda-forge/uv-feedstock

No, uv doesn't need python to run. python is not a runtime dependency of uv. uv can modify any python installation by installing pypi packages in there. It doesn't matter if the python installation is the system's python, a conda environment, or a virtual environment.

Because you may have several python installations like the system one and a conda env, uv has some heuristics to detect where your currently activated python env is. Then, when you run commands like uv pip install flask, it will install packages in that environment.

For conda environments, uv detects them through the CONDA_PREFIX env variable. So when I run something like:

conda activate my_env
uv pip install flask

uv will look at the CONDA_PREFIX env variable and find the python installation of the my_env conda environment and then installs flask there.

Now the problem with installing uv with pixi is that pixi always suppresses this CONDA_PREFIX env variable to point the uv environment. This happens inside ~/.pixi/bin/uv:

#!/bin/sh
export PATH="~/.pixi/envs/uv/bin:${PATH}"
export CONDA_PREFIX="~/.pixi/envs/uv"
"~/.pixi/envs/uv/bin/uv" "$@"

so when I do:

conda activate my_env  # conda activate exports a CONDA_PREFIX env variable
~/.pixi/bin/uv pip install flask

the CONDA_PREFIX variable doesn't point to the my_env conda environment anymore but points to export CONDA_PREFIX="~/.pixi/envs/uv" (from the bash file: ~/.pixi/bin/uv) and this breaks uv's python discovery: https://github.com/astral-sh/uv?tab=readme-ov-file#python-discovery

My sugesstion is not to add this line:

export CONDA_PREFIX="~/.pixi/envs/uv"

in the generated bash file when you see the user is installing uv with pixi which is what @wolfv suggested as well:

Hypothetically it might also be nice to have some shared configuration for all pixi users that covers some cases like this (either as annotations in the packages or as some external file).

[tdejager]

Yeah @183amir, I get what your are saying, and I agree that holds true for most use-cases. I'm just concerned about the use-case you want to use uv as a way to globally manage venvs, without having a global python interpreter installed. Without having python in the uv conda environment this won't work for the uv pip x commands, unless you a running in an activated environment in some way.

On the flip-side by keeping the CONDA_PREFIX this will shortcircuit detection in un-activated pixi folders without manually setting the VIRTUAL_ENV environment variable.

[183amir]

I'm just concerned about the use-case you want to use uv as a way to globally manage venvs, without having a global python interpreter installed.

If there is no global python interpreter installed, isn't it possible to install with pixi? like pixi global install python? Or something like pixi global inject uv python (if it gets implemented).

On the flip-side by keeping the CONDA_PREFIX this will shortcircuit detection in un-activated pixi folders without manually setting the VIRTUAL_ENV environment variable.

I am not sure what you mean here. If the CONDA_PREFIX is set to the uv global conda env, uv will always try to install packages there. What is the flip-side here?

If you guys are ok with adding an exception for uv to skip inserting the CONDA_PREFIX line in the generated bash file, I can work on a pull request.

[tdejager]

I am not sure what you mean here. If the CONDA_PREFIX is set to the uv global conda env, uv will always try to install packages there. What is the flip-side here?

There might not be globally activated env. The flip-side I mean is more in regards to if that variable is set uv will always use that python, so if there is a python included next to the uv env, it will prefer that even when running from a pixi environment for example. So if you global inject python next to uv, which would require the CONDA_PREFIX again, it will prefer that python. I'm just going from the documentation here.

If there is no global python interpreter installed, isn't it possible to install with pixi? like pixi global install python? Or something like pixi global inject uv python (if it gets implemented).

I don't think uv would find it. I believe this is the discovery function: https://github.com/astral-sh/uv/blob/e0ad649c7449b7f54c0053e409ac6e7bf18a6f68/crates/uv-toolchain/src/discovery.rs#L204C4-L204C40

[tdejager]

If you guys are ok with adding an exception for uv to skip inserting the CONDA_PREFIX line in the generated bash file, I can work on a pull request.

I'm fine with this, maybe initially as an option to global install? Not sure what @wolfv thinks.

[wolfv]

This could also be in a activation script in hte uv feedstock - that would say something like if $CONDA_PREFIX include ".pixi/bin" then unset CONDA_PREFIX fi.

Then we don't need to add any special rules into pixi. WDYT @183amir?

Thanks btw!

[183amir]

This could also be in a activation script in hte uv feedstock - that would say something like if $CONDA_PREFIX include ".pixi/bin" then unset CONDA_PREFIX fi.

That would still not work because uv needs access to the CONDA_PREFIX variable to find the currently activated env. unsetting it would not recover the previous CONDA_PREFIX variable.

BTW, I have the same issue with zsh now. I have installed zsh with pixi global install zsh and now I have a CONDA_PREFIX env variable set by default in my shell which breaks programs who rely on the CONDA_PREFIX env variable.

[Hofer-Julian]

I discussed this topic with @baszalmstra, and we identified that the main use case for setting CONDA_PREFIX are activation scripts of the package that might depend on it. We could stop setting CONDA_PREFIX if the package doesn't have an activation script. That would help with uv and zsh, neither of those packages has an activation script.

What do you think @wolfv?

[183amir]

The reason I proposed a symbolic link solution is that simply skipping activation scripts didn't fully address my issue. Another challenge I encountered was installing gridtk using pixi global install, which causes the tool to break. Specifically, the installed gridtk script (~/.pixi/bin/gridtk) ends up with a modified PATH environment variable, and this disrupts binary resolution when submitting jobs.

For instance, when I run:

gridtk submit ... python train.py

python is not selected from my current PATH; instead, it is sourced from the gridtk environment.

[Hofer-Julian]

This is really useful @183amir, thanks!

The PATH variable is also set in the script. Like the CONDA_PREFIX we could just not set it, if there's no activation script. Do you then still encounter problems that are fixed by symlinks, but not with this approach?

[baszalmstra]

@Hofer-Julian FYI that would not work in pixi shell, we rely on the PATH variable to find executables within the environment. But perhaps we can do something special for pixi global.

[Hofer-Julian]

But perhaps we can do something special for pixi global.

Yes, that's what I had in mind

[183amir]

The only other potential issue I can think of is that some installed binaries might become broken without a symlink, particularly those that rely on the current shell. This is because installed binaries typically run under /bin/sh, which is usually a symbolic link to /bin/bash. However, if you’re using zsh and the installed binary includes some kind of shell detection, it might not work correctly. That said, I haven't personally encountered this issue.