Multiple global environments

[jonashaag]

I regularly need multiple "global"/"non-project" environments that I can switch between.

Eg. I want to try something with Pandas 1 or Pandas 2 without having a specific project in mind.

Right now I have "pseudo" Pixi projects that I cd to and then use pixi shell.

I think what I want is Conda's -n option but not sure how that fits the "Cargo model" of implicit environment name/location

[baszalmstra]

I also have a few pixi projects on my hard drive that are just there to test some stuff in certain environments. I personally prefer to manage these myself instead of having a tool that creates named project "somewhere" ✨ . I like to be explicit and in control. I like that I know how to manipulate the filesystem and that those operations also map to managing pixi projects. I don't want to have to learn another set of commands to manage projects.

Nonetheless, I can understand that it can be a hurdle to create a temporary environment by first having to create a pixi project and then having to remember where they are.

Maybe you can help me understand the use case better. Could you elaborate a little bit on the following questions from your personal point of view?

• What is the difference between cding to a directory vs adding -n to a command?
• Would it help if instead of using "names" you could specify a pixi project directory explicitly?
• I assume you still have to remember the environment names when you use -n? Do you remember them? Does shell autocomplete help here? Do you use a tool? Something else?
• Would a cli/gui tool to manage your pixi environments be interesting?
• Would it help to create a temporary shell with a set of dependencies (e.g. pixi global shell -p python -p pydantic, or something similar to npx perhaps)?

I'm very interested in your (and others') point of view!

[hadim]

I would also be interested in that feature. It's true that one base default global env is already super useful but having the freedom to have multiple of them would be very convenient (for deps that simply dont work well in the same env). I like having those global envs not depending on any folder so they work in any terminal without having to do a cd.

[mjkanji]

Adding to that, I think it would be great if:

• pixi was global aware of all project-specific repos, thereby allowing you to achieve something like mamba activate some_env, where some_env is defined in a project folder somewhere. This would allow for very similar semantics/usage as (micro)mamba, allowing pixi to serve as a more complete replacement of (micro)mamba. This would, in essence, allow one to use a project env as a global env.

• If project-specific envs could be installed outside of the project directory by specifying a prefix in pixi.toml maybe? The use case here is that I currently use a cloud provider that creates ephemeral machines with free (block) storage, while the project files (i.e., code) are stored on a persistent EFS-like server that is mounted onto the ephemeral VMs. This is a typical managed-JupyterLab setup. As such, if the environment was saved under the root of my project (code directory), I'd be paying quite a bit for storage. My currently set up uses micromamba, where the environment is saved in the root directory in a Docker container (which uses the machine's storage), and the project files (code) are mounted on top. Having a way to link the external environment with the project would be a welcome addition.

Not sure if these should be separate issues -- if so, I'm happy to create new issues.

[baszalmstra]

@mjkanji Thanks for your feedback!

pixi was global aware of all project-specific repos, thereby allowing you to achieve something like mamba activate some_env, where some_env is defined in a project folder somewhere. This would allow for very similar semantics/usage as (micro)mamba, allowing pixi to serve as a more complete replacement of (micro)mamba. This would, in essence, allow one to use a project env as a global env.

I feel pixi should always act on the filesystem vs globally managed names. I like this better because it compartmentalized projects. For instance, by removing a project directory you are sure the entire thing has been removed from your machine and moving a project directory becomes trivial.

If you refer to a project from somewhere you should do so by the unique location on your system instead of by name. You can already do this with --manifest-path:

pixi shell --manifest-path /some/random/dir/pixi.toml

If project-specific envs could be installed outside of the project directory by specifying a prefix in pixi.toml maybe? The use case here is that I currently use a cloud provider that creates ephemeral machines with free (block) storage, while the project files (i.e., code) are stored on a persistent EFS-like server that is mounted onto the ephemeral VMs. This is a typical managed-JupyterLab setup. As such, if the environment was saved under the root of my project (code directory), I'd be paying quite a bit for storage. My currently set up uses micromamba, where the environment is saved in the root directory in a Docker container (which uses the machine's storage), and the project files (code) are mounted on top. Having a way to link the external environment with the project would be a welcome addition.

I totally get this! Cargo does something similar with the --target-dir flag and CARGO_TARGET_DIR. Could you create an issue for this?

[mjkanji]

I totally get this! Cargo does something similar with the --target-dir flag and CARGO_TARGET_DIR. Could you create an issue for this?

Created #255.

[jonashaag]

• What is the difference between cding to a directory vs adding -n to a command?

Typically I want the freedom to execute in the environment from any CWD, not just below the project root. Also, it's a bit more cumbersome because it's two commands. If used in scripts, you need to pushd/popd.

• Would it help if instead of using "names" you could specify a pixi project directory explicitly?

Yes!

• I assume you still have to remember the environment names when you use -n? Do you remember them? Does shell autocomplete help here? Do you use a tool? Something else?

I remember them and use shell history, not completion.

• Would a cli/gui tool to manage your pixi environments be interesting?

Not really; I'm fine with using ls and rm for that.

• Would it help to create a temporary shell with a set of dependencies (e.g. pixi global shell -p python -p pydantic, or something similar to npx perhaps)?

That would in fact be pretty awesome! But it's sort of unrelated IMO.

[twrightsman]

Part of why I really like pixi is that there is no global prefix assumption; I can download the binary and get started in a project directory without messing with my shell init scripts.

That being said, perhaps pixi could opportunistically discover global environments in a pre-defined, well-known path. On Linux this would be the XDG Base Dirs, like ~/.local/state/pixi/envs. Perhaps this could be overridden by an environment variable if a user really wants to, like PIXI_PREFIX.

[burdiyan]

I do prefer more compartmentalized approach of Pixi compared to Conda/Mamba. But the former support both approaches (global namespace, and project-specific). Would be great if Pixi supported both as well.

+1 for the approach @twrightsman recommended above.

[twrightsman]

@ruben-arts I read through the v0.19.0 changelog but I can't see the feature that fixes this issue. Could you share a snippet or demo that shows how to create multiple global environments?

[ruben-arts]

Sorry that was by mistake, I don't know how this happened

[ruben-arts]

This will be solved by the new implementation of the pixi global command. https://github.com/prefix-dev/pixi/milestone/9

I'll close this issue for now to avoid duplicate discussions.