pip-compile support for multiple Python versions

[john-bodley]

Would it be possible to add support for producing requirements.txt file which work for multiple Python versions via environment markers. The reason for the request is I'm working on a package which supports Python 2.7 and 3.4+, and we would like to include a requirements.txt file in the repo to ensure that development/test builds are predictable and deterministic.

Environment Versions

1. OS Type: Mac OS
2. Python version: $ python -V: Python 2.7.13
3. pip version: $ pip --version: pip 9.0.1
4. pip-tools version: $ pip-compile --version: pip-compile, version 1.11.0

Steps to replicate

In Python 2.7 flower has a requirement on futures, i.e.,
echo flower > requirements.in

pip-compile requirements.in
> ...
futures==3.2.0
...

however this package does not exist for Python 3.4 which means that pip install -r requirements.txt will fail in a Python 3.4 environment.

Expected result

It would be would be great if you could specify to pip-compile which versions the requirements.txt file should support, i.e.,

> pip-compile --envs py27,py34 requirements.in
...
futures==3.2.0; python_version == '2.7'
...

Note I'm actually creating this from setup.py and thus an alternative could be to filter out the packages which aren't explicitly mentioned in install_requires. I'm primarily using the requirements as part of tox, i.e.,

deps =
    -rrequirements

and though I could change this to be,

commands = 
    pip install -e .

which is really equivalent to:

commands = 
    pip-compile --output-file requirements.txt setup.py
    pip install -r requirements.txt
deps = 
    pip-compile

the process isn't deterministic (i.e., packages aren't pinned in setup.py), which is potentially problematic from a repeatability standpoint.

[vphilippon]

Hi @john-bodley,
pip-tools will not be changed to attempt to generate a requirements.txt files for other Python versions than the one of the current environment.
The reasons are pretty much covered in my recent comment here #635 (comment) as well as with this article: https://dustingram.com/articles/2018/03/05/why-pypi-doesnt-know-dependencies

In short, the dependency resolution results depends on way to many variable to try and fake them to obtain a reliable result.

What should be done is to execute a pip-compile on each target environment (ex: in each distinct virtualenv). With that in mind, pip-tools should be improved to interpret the environment markers in the requirements.in to generate a valid requirements.txt for the environnement in which it is executed. That would make the requirements.in reusable as an input file to pip-compile in each environment.

That's the plan so far regarding the multi-environment support.

[john-bodley]

Thanks @vphilippon for the explanation.