Support for --prefix

[FRidh]

Add support for --prefix. Note this is not the same as --destdir which was discussed in https://github.com/pradyunsg/installer/issues/58 and has been added.

Discussion on --prefix in https://github.com/pradyunsg/installer/pull/94#issuecomment-1013876999 and https://github.com/pradyunsg/installer/pull/66. I don't see anyway around needing --prefix in Nixpkgs.

A sysconfig scheme is static and doesn't provide the flexibility we need, that is, being able to install every package in its own prefix.

Alternatively, I could imagine reading a scheme from say a json or toml file when installing.

Something that could work for us:

var = {"installed_base": "$out", "base": "$out", "platbase": "$out", "installed_platbase": "$out"}
# Note there is no `sysconfig.get_default_scheme()`
sysconfig._expand_vars("posix_prefix", var)

[FFY00]

I think this should be fixed in the Python upstream, not this project, so I recommend you open a bug there asking for a way for vendors to specify dynamic sysconfig schemes (please add me to the nosy list: FFY00), and go from there.

Alternatively, this can be easily implemented in a Nixpkgs-specific install script.

[layday]

Would we be ok with addding an option for scheme names in the command line which would get passed to get_paths? Nix could define a var-less scheme directly under root and perform "staged" installs with the staging directory being the final install directory.

[pradyunsg]

Can someone explain how the two use cases are different, ideally with some illustrative paths as an example?

Alternatively, I could imagine reading a scheme from say a json or toml file when installing.

This is not the interface we'd take. Regardless, see #92 for the discussions on such a CLI.

At this point, you can (and, likely, should) write a small Python script that uses the API directly to do what you need. If there's more than one user who wants this functionality, we can explore adding that into this package. :)

[layday]

Can someone explain how the two use cases are different, ideally with some illustrative paths as an example?

If you have, say, a purelib path '{base}/lib/python{py_version_short}/site-packages', for a --destdir = '/foo' and a base = '/usr', which corresponds to the sys.prefix by default, the installation will be performed at:

/foo/usr/lib/pythonX.Y/site-packages

If you have a hypothetical --prefix = /bar, that would override the base path, so that the installation without a --destdir will be performed at:

/bar/lib/pythonX.Y/site-packages

In Nix, each package resides under a unique prefix, so that the Python a Python package is installed for will have a different prefix (aka base) from the package. Python will be installed at /nix/python-specific-prefix. But e.g. installer will be installed at /nix/installer-specific-prefix, and the same goes for every other package. Therefore, if Python's prefix is substituted for base with no possibility to override it, the resulting path will contain a reference to the Python prefix:

/nix/python-specific-prefix/lib/pythonX.Y/site-packages

... and Nix will need to shear it off somehow and prefix the package's actual prefix. This is made difficult by the fact that the prefix is an automatically-generated hash. What Nix have been doing is to install packages with pip install --prefix=/nix/package-specific-prefix but installer does not have an equivalent option.

[eli-schwartz]

Essentially, what is happening here is that unlike DESTDIR (which is used for installing into .deb or .rpm archives), Nix is actually installing to a different install scheme for every module. The install scheme includes e.g. the hash of the package name/version/contents.

What this means is you need a couple thousand different sysconfig schemes, all dynamically calculated.

I wonder if, instead, Nix could patch sysconfig with a nixpkg scheme, which detects via e.g. an environment variable whether it is in the nix package building environment, then adds that scheme using subprocess.run to run a Nix command that calculates the right absolute paths for purelib/platlib/etc.

[takluyver]

I'll also have a go at explaining destdir & prefix - I'm not disputing anything in @layday's explanation, but I think a bit more 'why' might be useful.

destdir (aka --root in pip) is useful if you're placing files whose paths will be different when they're actually used. E.g. you're preparing an SD card mounted at /mnt/sdcard - you might install a script at /mnt/sdcard/usr/bin/foo, but the system that will use it mounts the SD card as root, and the path becomes /usr/bin/foo.

prefix is more like a virtual environment - a relatively lightweight directory containing subdirectories like bin and lib. Things like Nix & Spack can put each package (not just Python things) in its own prefix, and combine them on the fly to make the necessary executables & libraries available.

I wonder if, instead, Nix could patch sysconfig with a nixpkg scheme, which detects via e.g. an environment variable whether it is in the nix package building environment, then adds that scheme using subprocess.run to run a Nix command that calculates the right absolute paths for purelib/platlib/etc.

I dare say Nix could, but I don't know why they would want to. Even without the option in the installer CLI, it would seem simpler for them to either stick with pip or write their own wrapper calling the installer Python API, so they can specify prefix directly rather than persuading sysconfig to give the answer they know they want. Patching and environment variables seems like a rather convoluted way to do a simple thing. (I'm not involved in the Nix project, though, so maybe they'll see it differently)

[layday]

I wonder if, instead, Nix could patch sysconfig with a nixpkg scheme, which detects via e.g. an environment variable whether it is in the nix package building environment, then adds that scheme using subprocess.run to run a Nix command that calculates the right absolute paths for purelib/platlib/etc.

There's not really any reason to go into all that effort if they are adding a new scheme. They can simply replace all base/platbase placeholders with / and pass --destdir as normal. All they'd need is a way to signal to installer that it should use the nixpkgs scheme.

[FFY00]

I wonder if, instead, Nix could patch sysconfig with a nixpkg scheme, which detects via e.g. an environment variable whether it is in the nix package building environment, then adds that scheme using subprocess.run to run a Nix command that calculates the right absolute paths for purelib/platlib/etc.

This would be my proposal. Currently, sysconfig lacks the "dynamicality" to do this, but that's what I think should be proposed to the Python upstream.

All they'd need is a way to signal to installer that it should use the nixpkgs scheme.

I'd recommend avoiding adding such an option until we get a path forward with the headers scheme key, which still depends on distutils right now. If really needed we could try to figure it out, but I just wanted to point out it that it's problematic and very tricky to handle properly, before anyone attempts to implement it.

[FRidh]

I wonder if, instead, Nix could patch sysconfig with a nixpkg scheme, which detects via e.g. an environment variable whether it is in the nix package building environment

Something like that was what I was thinking as well (see also https://bugs.python.org/issue43976#msg411657).

They can simply replace all base/platbase placeholders with / and pass --destdir as normal. All they'd need is a way to signal to installer that it should use the nixpkgs scheme.

I imagine this to typically work fine indeed.

As suggested, I think it will be easier in the end for us to write our own script using the installer API in the end. Of course, such script will also be needed by some other distro's such as Guix and Spack if they were to migrate from pip to installer. Maybe something could improve in sysconfig, but it seems convoluted to me, and realistically speaking it won't happen for 3.12 anyway either.

[lazka]

In MSYS2 we also install to a different prefix compared to the installed Python, so this would also be helpful to us.

Our Python installation is relocatable, so our packages are too (just like wheels)

[mgorny]

Actually, I think this would be helpful outside of Nix too. Just imagine you're running installer inside a venv. Since venv overrides prefix, the package files are installed in the venv-alike layout, so in the end you can't normally use venv to prepare install trees for the system.

Let's say I'm running a virtualenv at /tmp/venv, installed installer there and now want to prepare a install tree for the real system. I run python -m installer --root=/tmp/z and I get... /tmp/z/tmp/venv/lib/python3.10.... What I need is to force /usr prefix, so I'd get /tmp/z/usr/lib/python3.10....

[mgorny]

Though I suppose a hack like python -c "import sys; sys.prefix = sys.exec_prefix = '/usr'; from installer.__main__ import main; main(sys.argv[1:])" works too ;-).

[eli-schwartz]

Or using /usr/bin/python -m installer so the venv (which is just another python earlier in your path) isn't used.

Or if you must install installer itself in a venv (why? this is not something Gentoo itself should need, is it just a shower thought?) then being able to do /tmp/venv/bin/python -m installer --interpreter /usr/bin/python solves this just like any other "different install interpreter and installer-tool interpreter" case.

[mgorny]

/usr/bin/python -m installer

But then it wouldn't find the installer inside the venv.

[eli-schwartz]

Well I mean yes, but... That's why I suggested the correct solution would instead be --interpreter rather than --prefix, at least for cases other than Nix.

But you still haven't explained why it's important to Gentoo that you can install the installer-tool into a venv and use it to install wheels outside of the venv.

[mgorny]

We're installing to a temporary venv to run tests in, then we're stripping the venv bits and transferring the result to the filesystem.

[mgorny]

In either case, this looks really trivial to do and I'm working on a PR where we can discuss this further.

[pradyunsg]

In either case, this looks really trivial to do and I'm working on a PR where we can discuss this further.

I'd prefer to keep discussion of "should we do this" in the issue, keeping the PR purely for implementation-related concerns. And, uhm, "this is trivial to do" isn't sufficient rationale to do something like this (not that that's the only rationale provided here).

[pradyunsg]

That's why I suggested the correct solution would instead be --interpreter rather than --prefix, at least for cases other than Nix.

FWIW, this is how I'm leaning as well.

We're installing to a temporary venv to run tests in, then we're stripping the venv bits and transferring the result to the filesystem.

Is Gentoo currently using setup.py directly? If not, then you're using pip and "install this wheel in this venv" is basically a drop-in replacement with installer -- albeit with different code backing that logic.

A better way of phrasing this: So, if installer is able to install into a different Python interpreter, that would satisfy your usecase, correct?

from installer.__main__ import main; main(sys.argv[1:])

I'll take this as a good reminder to change the name to _main to reflect that it's not public API. :)

Edit: Done in fad2894

[pradyunsg]

In MSYS2 we also install to a different prefix compared to the installed Python, so this would also be helpful to us.

Could you elaborate a bit on why this would be helpful? I'm still very much at a point of trying to gather a better understanding of what workflows redistributors have, to figure out what are good/reasonable ways to support them in this package.

[FFY00]

FWIW, this is how I'm leaning as well.

Having had to deal with a bunch of issues related to this, I'd also very much prefer that in this package.

[mgorny]

A better way of phrasing this: So, if installer is able to install into a different Python interpreter, that would satisfy your usecase, correct?

I'm sorry but I'm not sure if I understand what you're saying. If you mean that when started from inside a venv, installer would have paths as if it had been started via /usr/bin/python*, then yes, I think so.

[FFY00]

No, it would have the paths of the venv (eg. my-venv/lib/python3.10/site-packages).

[pradyunsg]

@mgorny could you remove the reference to the issue from the commit message? It’s unnecessarily causing spam in this discussion and it’s generally not a good idea to mention an issue when you have a corresponding PR open.

[mgorny]

@mgorny could you remove the reference to the issue from the commit message? It’s unnecessarily causing spam in this discussion and it’s generally not a good idea to mention an issue when you have a corresponding PR open.

Sorry. Done that now.

[adamjstewart]

Spack developer here. Just want to +1 this idea. Like Nix and Guix, Spack installs every single software package to a separate installation prefix. This allows us to, for example, install 3 different versions of numpy with 4 different compilers and 5 different BLAS/LAPACK providers. We then either add these directories to PYTHONPATH or create a single environment via symlinks. Right now, we install all Python libraries with pip install --prefix but we would love to use installer instead someday.