Add an option for pip to manage a different "environment"

[pfmoore]

What's the problem this feature will solve?

At the moment, pip can only fully¹ manage the current Python environment. This makes it hard to have environments without pip installed (for example, when preparing a zipapp, or an embedded Python installation).

Being able to specify a target "environment" (in the sense of a set of paths like sysconfig scheme) which pip can manage, would allow a single shared pip to be used for multiple Python environments.

Describe the solution you'd like

Add a global option to pip that sets the "environment" to be managed. The UI for this option is yet to be decided - in the most explicit case, values are needed for the 5 sysconfig paths that pip can install data into, but shortcuts for common cases would be important for a good user experience.

All pip commands will use the specified environment.

Alternative Solutions

• The existing --target, --root and --prefix options all attempt to handle this, but they lack support for upgrades and uninstalls, as well as not being integrated with informational commands like list.
• Running pip with the Python environment's python executable works, but only when the target environment is a virtual environment. This does not cover cases like preparing a zipapp, or installing into the library area of an app that embeds Python.
• Rather than having a UI for selecting a scheme, it would be possible to have a (simple) programmatic API to allow people to set a scheme, and then invoke pip¹. But that would make "manage another environment" inaccessible from the fundamental python -m pip command.

Additional context

This is closely related to #4575. A lot of the discussion on that issue covered handling environments that "shadow" each other (user shadowing system, and --use-system-site virtual environments). This issue explicitly avoids that problem, focusing on the case of a single standalone environment, with shadowing being left as an issue for how the user sets up their sys.path.

This proposal doesn't stand a chance of working with the "legacy" direct setup.py install method (for editables or normal installs), as that method delegates installation to setuptools, and it is impractical to force setuptools to conform to the scheme selected in pip¹. So maybe it's time to finally pull the plug on that method? What remains to do to make that happen? At an absolute minimum, we should make the legacy install method fail if a custom scheme is set by this option.

Testing will be a possible issue. In theory, we should test as much of pip's functionality as possible with a custom scheme, as well as with the default scheme. But that will be a fairly massive increase in our already large test times². I feel like it would be worth reorganising the test suite to have a "core functionality" section that's run on a matrix basis (normal, zipapp, custom scheme, ...) with the remaining functional tests being normal scheme only (but if we do that, are we just crossing our fingers that there's no edge cases?) I don't feel anything like confident enough in my understanding of our test suite to tackle something like this, though. For now, I'll probably just add "custom scheme" tests as I go, ignoring the test runtime issue, and leave refactoring the test suite to someone with more expertise...

Code of Conduct

• I agree to follow the PSF Code of Conduct.

Footnotes

1. The --target, --root and --prefix options only apply to pip install. ↩ ↩² ↩³

2. Particularly if we also add testing the zipapp version, which has the same sort of "everything should work the same" problem. ↩

[uranusjr]

This actually don’t necessarily need to depend on #4575? (Although it’d of course make this much easier to implement.) With pip being distributable as a standalone application, the flag could be implemented by simply packaging pip into that, and create a subprocess to do the actual work against the specified interpreter.

[pfmoore]

This functionality doesn't require there to be a specified interpreter. It would act as a (more complete) replacement for --target, for example. Also, to be clear, I'm not suggesting that this depends on #4575 but rather that there's a lot of discussion under that issue that may be relevant here as well (the bits that aren't about schemes shadowing each other).

As a straw-man proposal, I'm suggesting a global pip option --scheme platlib=/path/to/platlib,purelib=/path/to/purelib,... All the valid scheme locations should be specified, and the paths can be totally unrelated to one another, if you want. In reality, we'd need a better UI, but that will do for an initial implementation, to prove the approach.

[pradyunsg]

Quick set of thoughts before I move over to my work laptop:

• We don't need to repackage pip for this at runtime -- putting it on the import path at the start and running with that is good-enough. And, we have this now -- with the _run-pip script or whatever that's called.
• I'm strongly opposed to exposing exact scheme paths via the CLI. That's a discussion we've had over on installer (Add CLI installer#66, Add simple, explicit CLI to install a .whl file installer#92, CLI to install to Python running installer installer#94). To me, the conclusion was that the potential for pain, caused by someone who doesn't fully know what they're doing, is way too big to allow callers to do that. It's too big of a footgun, that starts by being pointed at your leg.

Beyond that, IIUC, there's three "points" of coupling with the currently running interpreter:

1. The installation paths, which are also affected by the options passed to pip.
2. The build environment and the Python executable being used in it.
3. The "tags" + markers environment being used to select the distributions in the dependency resolution process.

I think, to make pip work correctly with the various flags that we have, we should consider what our flags do on this front today. I believe we don't have anything that affects 2. We have flags that should affect 3 and don't. We also don't have any way for a user to specify all of these, which we probably should.

A --python would require us to immediately subprocess call with the provided Python interpreter, but it'd give us all of this information "for free", by doing a short-circuited call to the Python that actually needs to be used. The rest of the code would not need to know that there's an eager subprocess call happening.

[pfmoore]

OK, I got over-excited by the results of my investigations, and as a result, generalised my findings way beyond what they justified.

@pradyunsg you're absolutely right, a --python option that simply works out what files to install using the current interpreter and then dumps them in the target interpreter's environment would be incredibly broken. In my defense, I was thinking of "use the base Python to install stuff in a virtualenv built from it", which is basically about the only case where that approach stands any chance. So yes, I completely agree, we should be implementing --python by re-invoking pip with the target interpreter.

On the other hand, the Scheme mechanism can be used to create a more fully functional version of --target (and --prefix and --root) which supports the full set of pip commands (pip install --upgrade, pip list --outdated, pip uninstall, pip freeze, etc). That's what my prototyping demonstrated, and is actually the use case I'm personally more interested in, which is why I was focused on the "set the target paths" mechanism¹.

Apologies for the confusion. I find issues a fairly clumsy way of handling the sort of "exploring the design space" work that I'm currently doing - but I don't know of any other, better way of sharing my findings (and unfortunately for the rest of you, I work better by sharing progress as I go along 🙄)

So to summarise:

1. --python should be implemented by re-invoking the current pip with the target interpreter.
2. An improved --target can be implemented by setting the global scheme up front²

Both of these seem like reasonable new features to add - does anyone have any reservations regarding either of them?

Footnotes

1. I still quite like the idea of having a mechanism to set the paths individually, but more for the "pip as a tool run by other tools" use case - if we had a programmatic API, it would be an API, but as we don't, it's a CLI option... As it's for tools to use, it doesn't need to be particularly user-friendly 🙂 ↩

2. Note to self, the legacy setup.py install won't respect the scheme, so maybe if the global --target is specified we should force PEP 517 mode? ↩

[pfmoore]

--python should be implemented by re-invoking the current pip with the target interpreter.

Draft PR for this at #11320. @pradyunsg's work on __pip-runner__.py made this beautifully simple 🙂

[pfmoore]

So to summarise:

1. --python should be implemented by re-invoking the current pip with the target interpreter.
2. An improved --target can be implemented by setting the global scheme up front

Both of these seem like reasonable new features to add - does anyone have any reservations regarding either of them?

We now have a --python flag for pip. As far as item (2) is concerned, I've raised #11366 to focus on that specifically. So I'll close this issue as there's nothing futher needed here.