#206: Preserve environment markers from requirements.in

[barrywhart]

Replaces #459

• piptools/scripts/compile.py
  • When calling OutputWriter.write(), pass along the markers from requirements.in
• piptools/util.py
  • format_requirement() -- add support for optional markers
• piptools/writer.py
  • Add marker parameter to OutputWriter.write(), pass it to piptools.util.format_requirement()
• tests/test_writer.py
  • Test for the new behavior

[adamchainz]

As per your previous comment, I would add a comment to the source here why you're transforming the quotes. You could also make it obvious with a if sys.version_info... for the affected Python version

[barrywhart]

I will revisit this. A better solution might be to use double quotes in the environment marker -- I suspect these will remain double quotes in both 2.7 and 3.5. That would eliminate the need for the non-obvious replace() operation.

[barrywhart]

I have cleaned this up -- the test now uses double quotes, which work consistently on Python 2.7 and 3.5.

[davidovich]

I think this will be good for the 1.9 release.

[suutari-ai]

Is this really something that pip-compile should do? Shouldn't the compiled requirements.txt target a single environment rather than having conditional dependencies for several different environments? (Environment = Python version, platform (linux/win/..), etc. which can be conditional with the markers)

• E.g. if I have a conditional requirement foobar; python_version < "3.0" in my requirements.in and I compile that with Python 2, should it still generate the foobar==ver dependency to the txt file (with the marker)?
• What if some package foo has conditional dependencies like baz2; python_version < "3.0" and baz3; python_version >= "3.0", should pip-compile be able to to add both packages with their pinned versions to the generated txt?

What I'm thinking here is that, if the requirements.txt should be compiled in a way that it supports more than one environment, then the compilation should be able to collect all possible environment variations from the source dependencies and pin the versions and use markers accordingly. This will get very complicated. I thought pip-compile is about locking down the specific versions and specific packages and conditional entries in the generated file doesn't feel like it's in line with that.

[barrywhart]

I think your question contains a misconception: When pip-compile "compiles" a requirements.txt file, the Python version and OS used to execute pip-compile has no impact on the output. The environment markers are simply passed through. Thus, the requirements.txt file will simply contain a line like:

foobar==1.2.9; python_version < "3.0"

The environment marker is not interpreted until package installation time. Thus the requirement.txt file is potentially compatible with any version of Python.

This feature has proven very useful to me so far, and several others have requested it, too. The target use case is when there is a library that simply doesn't work on certain interpreters or platforms. Two examples:

• functools32 is a 2.7 backport of Python 3's built-in functools library
• PyPy does not support gevent, thus any packages such as locustio which require it do not work.

Hence, an application might contain the following lines in requirements.in:

functools32 ; python_version=='2.7'
locustio==0.7.5 ; platform_python_implementation != 'PyPy'

[pjenvey]

str(marker)'s redundant here, format's {} does it for you

[barrywhart]

Will remove.

[pjenvey]

I don't think this comment applies?

[barrywhart]

Probably copy-paste. Will review and fix or remove.

[suutari-ai]

I think your question contains a misconception: When pip-compile "compiles" a requirements.txt file, the Python version and OS used to execute pip-compile has no impact on the output.

Do you mean that your PR changes this somehow?

At least currently it makes a big difference. Here's an example:

$ . venv-py2/bin/activate
(venv-py2) $ pip-compile --version; python --version
pip-compile, version 1.8.0
Python 2.7.12
(venv-py2) $ cat requirements.in 
ipython
(venv-py2) $ pip-compile requirements.in -o requirements-py2.txt
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --output-file requirements-py2.txt requirements.in
#
appdirs==1.4.3            # via setuptools
backports.shutil-get-terminal-size==1.0.0  # via ipython
decorator==4.0.11         # via ipython, traitlets
enum34==1.1.6             # via traitlets
ipython-genutils==0.2.0   # via traitlets
ipython==5.3.0
packaging==16.8           # via setuptools
pathlib2==2.2.1           # via ipython, pickleshare
pexpect==4.2.1            # via ipython
pickleshare==0.7.4        # via ipython
prompt-toolkit==1.0.13    # via ipython
ptyprocess==0.5.1         # via pexpect
pygments==2.2.0           # via ipython
pyparsing==2.2.0          # via packaging
scandir==1.5              # via pathlib2
simplegeneric==0.8.1      # via ipython
six==1.10.0               # via packaging, pathlib2, prompt-toolkit, setuptools, traitlets
traitlets==4.3.2          # via ipython
wcwidth==0.1.7            # via prompt-toolkit

# The following packages are considered to be unsafe in a requirements file:
# setuptools                # via ipython
(venv-py2) $ deactivate
$ . venv-py3/bin/activate
(venv-py3) $ pip-compile --version; python --version
pip-compile, version 1.8.0
Python 3.5.2
(venv-py3) $ pip-compile requirements.in -o requirements-py3.txt
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --output-file requirements-py3.txt requirements.in
#
appdirs==1.4.3            # via setuptools
decorator==4.0.11         # via ipython, traitlets
ipython-genutils==0.2.0   # via traitlets
ipython==5.3.0
packaging==16.8           # via setuptools
pexpect==4.2.1            # via ipython
pickleshare==0.7.4        # via ipython
prompt-toolkit==1.0.13    # via ipython
ptyprocess==0.5.1         # via pexpect
pygments==2.2.0           # via ipython
pyparsing==2.2.0          # via packaging
simplegeneric==0.8.1      # via ipython
six==1.10.0               # via packaging, prompt-toolkit, setuptools, traitlets
traitlets==4.3.2          # via ipython
wcwidth==0.1.7            # via prompt-toolkit

# The following packages are considered to be unsafe in a requirements file:
# setuptools                # via ipython

My point is: If you want the same requirements.txt to work for different environments, then pip-compile should collect all the possible environment markers that every package in the dependency tree has and take those into account. These env markers should then be combined and the generated file should have those env markers added. I don't know how easy that would be, but I bet generating a different requirements.txt files for each environment is much easier.

[barrywhart]

Interesting, thanks for pointing out how the output differs. I hadn't noticed that.

In spite of the behavior you point out, this PR still implements the feature in a useful way. Your example is similar to how I am using this feature in my own work.

If I read the output correctly, the only difference between the two outputs is this package for Python 2.7:

backports.shutil-get-terminal-size==1.0.0

To deal with this, simply add the following line to requirements.in:

backports.shutil-get-terminal-size ; python_version < "3.0"

With this addition, pip-tools creates a requirements.txt file that works on both Python 2 and Python 3.

I do appreciate the comments and suggestions here, but I don't think they make a case against merging this PR. The comments fall into two categories:

• Let's produce a different requirements.txt for each environment. While this might be easier, it does not address the original request (add support for environment markers in requirements lines for pip-sync #206), which happens to be what I need as well. Some build environments (notably TravisCI) automatically install requirements.txt; they don't expect there to be a different file per environment.
• Let's implement something more complex. These arguments seem to come from an abstract perspective, i.e. problems that could occur, not from real-world problems like those discussed in the original issue.

Please, let's review this PR in the context of the original issue from 18 months ago, which several people have weighed in on and agreed is useful. Once this feature is in pip-compile, if it needs additional work to handle new requirements that arise, we can revisit it then. Note this feature has no impact on the behavior of pip-compile unless someone explicitly adds environment markers to requirements.in. So this feature solves real-world problems that have affected several people, and it's harmless if you don't want or need to use it. What's not to like?

[adamchainz]

If I read the output correctly, the only difference between the two outputs is this package for Python 2.7

Actually there's also enum34, pathlib2, and scandir on Python 2. It would have been useful if diff was used in the example.

However I still agree with you @barrywhart , I think this feature can be merged as-is and improved down the line. It's not a huge burden to copy some packages with environment markers into requirements.in to avoid having two files for two systems.

[suutari-ai]

Note this feature has no impact on the behavior of pip-compile unless someone explicitly adds environment markers to requirements.in. So this feature solves real-world problems that have affected several people, and it's harmless if you don't want or need to use it. What's not to like?

I think it should be at least possible to toggle between preserving the environment markers from the in-file or to resolve them. Suppose you're using the "different file for each env" flow which I suggested and have some env markers in the in-file. Then you'd expect each output file to have different contents (e.g. requirements-py2.txt only having the py2 deps and requirement-py3.txt having py3 deps): each file having only the concrete dependencies for that env.

[suutari-ai]

How does this interact with generate-hashes option btw.?

I've previously had some problems when I used a requirements.txt file with the generated hashes and tried to install those requirements with different Python version. The txt file was for Python 3.4 and I installed it with Python 3.5, so it wasn't even a major version difference. Having the same txt file used for more than one env should then probably have hashes of each env too or otherwise the install will fail to hash mismatch.

Yeah... this is also another story and this feature will still be useful when not using hashes, but should the --generate-hashes option be disabled while using this feature? (Since these two don't combine well.)

[barrywhart]

I'm not familiar with generate-hashes. From your description, it sounds like that option has issues or limitations already.

I think I would prefer someone document the limitations of generate-hashes in a separate PR. Trying to have one option automatically disable another seems tricky and confusing.

[suutari-ai]

I wasn't suggesting anything fancy for this. Maybe something like this could work:

# in scripts/compile.py
@click.option(...)
@click.option('--preserve-env-markers', ...)
def cli(verbose, ..., generate_hashes, ..., preserve_env_markers, ...):
    ...
    if preserve_env_markers and generate_hashes:
        raise raise click.BadParameter(
            "--preserve-env-markes is not compatible with --generate-hashes")

[barrywhart]

Would anybody else like to weigh in on the --generate-hashes proposal above? I'm open to it but have no strong opinion yes or no.

[barrywhart]

Ok, I have addressed the two code review comments.

[davidovich]

Thanks @barrywhart !