Proposal: pip-specific environment variable expansion in requirements and constraints files

[uranusjr]

What's the problem this feature will solve?
PEP 508 URL specification has been proposed (partly) as the replacement to dependecy_links in setup.py. This however causes problems for artifact vendoring since the file:// protocol does not provide a way to represent relative paths and a project’s requirements.txt must hard-code the project root, or use the pip-specific path format, which is not always viable.

Describe the solution you'd like
The proposal is to introduce pip-specific environment variables that are populated only during the parsing phase of requirements and constraints files, so users can use them in PEP 508 URL specs. These environment variables should have the prefix PIP__ (two underscores to avoid conflicts with the command argument variables), and are guarenteed to use the POSIX path representation (i.e. forward slashes, even on Windows). Two such environment variables are proposed for initial addition:

• PIP__WORKING_DIRECTORY expands to the direcotry where the pip command is run in, i.e. Path.cwd(). This is the same as the POSIX PWD, but proposed for platform-independent projects.
• PIP__CURRENT_FILE expands to the requirements or constraints file that contains the requirement line. This makes it easy to e.g. put the requirements.txt and vendored artifacts in the same directory, and refer to an artifact with package @ file://${PIP__CURRENT_FILE}/../package-1.0-py3-none-any.whl.

Alternative Solutions

• Allow paths in PEP 508. I don’t really like this (filesystem path is a hairy thing to specify in a cross-platform way), and have a feeling others may share the same view.
• Propose a pip wrapper that automatically generates a requirements.txt from a template. But requirements.txt is underspecified and difficult for a third-party tool to support. Ican develop for only the scenarios I personally need, but it is difficult to suggest it to other people.

Additional context
This would also serve as a solution to #8765. Users can use these environment variables instead of relative paths to avoid the quirky resolution logic. We can even consider deprecating relative paths altogether if this gets adopted well.

[sbidoul]

I may miss something but it looks like relative URLs, if implemented properly as described in #8765, would do the job?

Relative URLs is a fairly well understood concept (in html at least) which fits quite naturally with requirement files.

Of course it is yet another transition to manage, but I'd say it's not worse than deprecating relative requirements.

[uranusjr]

relative URLs, if implemented properly as described in #8765, would do the job?

It would do one half of the job. The main problem I have with #8765 is that no matter which logic we choose, people depending on the other resolution logic can no longer do what they’re doing, and I can see both ways being useful in certain workflows. We probably should unify the relative path resolution logic for consistency, but IMO we should also offer an alternative if we do.