Environment markers do not carry out to transitive dependencies

[tuukkamustonen]

As in title.

Environment Versions

Ubuntu 14.04, pip-tools 1.9.0, pip 9.0.1, python 2.7 + 3.4

Steps to replicate

Imagine some package 'foo` that requires:

enum34; python_version < '3.4'

Require foo in requirements.txt:

foo

With python 2.7, when you run pip-compile requirements.txt...

Expected result

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --output-file requirements.txt requirements.in
#
enum34==1.1.6; python_version < '3.4'           # via foo
foo==1.0.0

Actual result

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --output-file requirements.txt requirements.in
#
enum34==1.1.6           # via foo
foo==1.0.0

So environment marker is not carried on.

If you run with python3.4+, whole enum34 is omitted.

[suutari-ai]

It's not actually only the environment markers that will affect the dependencies. Also different distribution packages for the same requirement might have different dependencies.

An example of this is provided in issue #558: django-auth-ldap==1.2.15 has three different packages:

• django_auth_ldap-1.2.15-py2-none-any.whl requiring django and python-ldap>=2.0;
• django_auth_ldap-1.2.15-py3-none-any.wh requiring django and pyldap; and
• django-auth-ldap-1.2.15.tar.gz requiring django and either python-ldap>=2.0 or pyldap depending on which python version is used to run the setup.py.

In ideal world everyone would use environment markers, but in reality there are many distributions in PyPI which have different dependencies for the different packages of a release.

[vphilippon]

Carrying over the environment marker would't be easy, because that's handled directly by pip and we simply get the resulting dependencies after their evaluation.

Even if we'd manage to do this in a maintainable way, we'd have to compute the dependencies for each condition branching from those markers for the output to be exact.

So unfortunately, the generated requirements.txt are not guaranteed to be "environment independent", and IMHO, trying to achieve this would be a really hard task that would require a lot of testing.

[tuukkamustonen]

I feel the problem. I am new to pip-tools, and I thought #460 was supposed to make builds environment-agnostic. But it works on main level only then, so the implementation is like half-way there?

[vphilippon]

#460 was really only about keeping explicit environment markers from the requirements.in, not being environment-agnostic (you can see the discussion of PR, suutari-ai covers it well). It's only a feature to deal with simple well known cases with no subdependencies where it is safe to simply keep the environment markers.

As an example, you could add enum34; python_version < '3.4' to your requirements.in, as you know it will be one of your dependencies in your example, and it's risk-free as enum34 has no dependencies, so the resulting requirements.txt should not vary. This gives you an environment-agnostic requirements.txt for this simple case, which could not be done before that PR.

[barrywhart]

I was the author of the other change (#460). And that's true, it was a tactical improvement not intended to automatically address all possible issues. I think that other change may still help with this issue if you can identify and add the "problem" requirements to your requirements.in file along with an appropriate environment marker.

In essence, you would be working around the lack of a complete solution by manually doing what the tool does not do -- marking the problematic transitive dependencies. I'm curious to know if this helps in your situation. As others have mentioned, the potential true solution is more complex, and I don't believe anyone has worked out a detailed design for how that might work and whether it'd be useful to people, i.e. would the cure end up being worse than the disease. :-)

[tuukkamustonen]

Manually investigating / marking the transitive dependencies with their environment markers would be too cumbersome. It's just not really a workaround, imo.

Anyway, I have no urgent need for what this ticket is about - I just expected transitive dependencies to be also tackled, but they weren't so raised a ticket to be enlightened :).

I tend to develop on the same OS and major version of python (e.g. 3.5.x) as what my servers are running. So locking not only the versions but also losing the environment markers is not a problem. I would assume this applies to most developers out there?

Utility libraries/SDKs need to be compatible with multiple environments, and cannot lock versions, so there's no use for pip-tools in such projects anyway.

Of course, locking development / test dependencies in any project can also be useful. And currently the lock files pip-tools produces are not "safe" to use if developers work in different OS / python versions. Is the recommended way to tackle that to develop on similar environments?

[barrywhart]

I'm thinking you only do this for the problematic ones. No problems, no work to do. :-)

IMHO, developers should work on Vagrant or Docker so the environments are similar. That's just me speaking as a developer; I'm a one-time contributor to pip-tools so I can't speak for the project.

Open source can be a challenge. My employer doesn't mind me contributing, but they don't encourage me to contribute in areas beyond those useful to my own work.

Whenever this topic (complicated dependencies) comes up, there are lots of suggestions, but it never quite seems like anybody really has to have it. And with the complexities involved, it could easily be several days or even a few weeks of work to design, code, and test. Realistically, I don't think it's going to happen until it does become a hard requirement for someone.

[tuukkamustonen]

I have no urgent need for what this ticket is about

Actually, a (pretty standard) use case where this is needed: When you support multiple python versions for a library, and run tests under tox (or jenkins), then you cannot simply have something like this in tox.ini:

commands = \
    pip install -r requirements.txt

As requirements.txt contains the compiled dependencies against specific platform/environment, and probably doesn't work under something else (e.g. 2.7 vs 3.6).

[davidovich]

@tuukkamustonen What prevents you from outputing frozen requirements per environment and use that as an input to the pip install (with the tox matrix reference) (did not test) ?:

commands = pip install -r requirements_{env}.txt

[tuukkamustonen]

It's just that producing/maintaining those files would be somewhat cumbersome. I would need to switch between virtualenvs and build the files for each env by hand.

Some sort of tooling would be needed to make that sensible. There was a suggestion in #635 (comment).

Thought, maybe a simple for loop could provide a good start, something like:

for ENV in 27, 34, 35, 36
    pyenv activate project${ENV}
    pip-tools compile -o requirements.txt requirements.in
end

[AndydeCleyre]

Actually, a (pretty standard) use case where this is needed: When you support multiple python versions for a library, and run tests under tox (or jenkins), then you cannot simply have something like this in tox.ini:

commands = \
    pip install -r requirements.txt

As requirements.txt contains the compiled dependencies against specific platform/environment, and probably doesn't work under something else (e.g. 2.7 vs 3.6).

Now that 5.0.0 is out, using pip install -r requirements.txt is achievable, if still a bit cumbersome, as incompatibly marked requirements will be safely ignored; see my comment here.

[mitchhentges]

Hey 👋 I encountered this exact problem, and I've written a little friendly alternative to pip-compile that successfully carries out transitive dependencies: see pip-compile-cross-platform.