pip-compile fails on unavailable irrelevant constraint

[clslgrnc]

pip-compile tries to find a match for all constraints, event those that are not a requirement and come from a constraint file.

As a result if a constraints file defines a constraint on a package that is unavailable (be it a typo, a private lib on a currently unavailable repository, or a deprecated version no longer available) pip-compile won't work, even for requirements.in that will not need that package.

Steps to replicate

requirements.in:

-c constraints.txt

constraints.txt:

invalid-dependency~=0.0

Expected result

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile requirements.in
#

Actual result

Could not find a version that matches invalid-dependency~=0.0 (from -c constraints.txt (line 1))
No versions found

This should be fixed by #1175

[atugushev]

Hello @clslgrnc,

Thanks for the issue an the fix! Looks like pip doesn't resolve constraints too if they are not presented in requirements.txt.

[jamescooke]

@clslgrnc Thanks for this report - I think I've run into this today, maybe you would consider adding the following as a test case? Or could you confirm that your fix in #1175 will solve this issue?

The reason I'm asking is because the msgpack constraint is on a "real" package, but the version of < 0 is unfulfillable, which is slightly different to the example in your report above.

The second reason I'm asking about this is because it might open the door to another way to solve #333 < I'm currently trying to install firebase-admin keeping msgpack-python as a dependency and not updating to the renamed msgpack.

Example files

requirements.in:

-c constraint.txt
pip-tools

constraint.txt:

msgpack < 0

Current behaviour

With pip-tools==5.2.1 on Python 3.7, pip-compile -v fails:

                          ROUND 1                           
Current constraints:
  msgpack<0 (from -c constraint.txt (line 1))
  pip-tools (from -r requirements.in (line 2))

Finding the best candidates:
Could not find a version that matches msgpack<0 (from -c constraint.txt (line 1))
Tried: 0.5.0, 0.5.1, 0.5.2, 0.5.3, 0.5.4, 0.5.5, 0.5.6, 0.6.0, 0.6.0, 0.6.1, 0.6.1, 0.6.2, 0.6.2, 1.0.0, 1.0.0
Skipped pre-versions: 1.0.0rc1, 1.0.0rc1
There are incompatible versions in the resolved dependencies:

☝️ That's the end of the output - there are no resolved dependencies listed.

Expected behaviour

pip-tools doesn't need msgpack, so we don't care that there isn't a version < 0. Compile should pass with the usual output.

[atugushev]

@jamescooke have you tried #1175?

[jamescooke]

@atugushev No I've been lazy and just typed 😞

[clslgrnc]

It works with #1175: parsing the constraint does not fail and it does not try to resolve msgpack<0 since it is not refered to by pip-tools.
I am not sure how it could help solve #333 though.

[jamescooke]

@atugushev I've given #1175 a go and it works well for the example above 🙌 - thanks @clslgrnc for your work on this 👍

Now using #1175, I can create an impossible constraint against an existing package with < 0. This prevents that package ever being included in the compiled requirements - and the fix in #1175 means that pip-compile will pass when the banned package is not included in the requirements.

In this case, constraining with msgpack<0 (as per the example above) means that no compiled requirements can ever include msgpack. The newly named msgpack causes a conflict with the originally named package msgpack-python so until it can be resolved in the project I'm working on, it needs to be prevented from being installed. I think this is a similar issue to what's described here #333 (comment) with enum and enum34.

Example files

constraints.txt:

msgpack<0

requirements.in:

-c constraints.txt
firebase-admin<4

pip-compile gives an error:

Could not find a version that matches msgpack<0,>=0.5.2 (from -c constraints.txt (line 1))               
Tried: 0.5.0, 0.5.1, 0.5.2, 0.5.3, 0.5.4, 0.5.5, 0.5.6, 0.6.0, 0.6.0, 0.6.1, 0.6.1, 0.6.2, 0.6.2, 1.0.0, 1.0.0
Skipped pre-versions: 1.0.0rc1, 1.0.0rc1                                                                 
There are incompatible versions in the resolved dependencies:                                            
  msgpack<0 (from -c constraints.txt (line 1))                                                           
  msgpack>=0.5.2 (from cachecontrol==0.12.6->firebase-admin==3.2.1->-r requirements.in (line 2)) 

This is great because it's what I'm looking for - msgpack is banned.

At this stage, if the resolver had increased search functionality then it might find the solution - but I'm not complaining 😊 ...

We can get compilation to pass by pinning one of the sub-dependencies:

-c constraints.txt
firebase-admin<4
cachecontrol<=0.12.4

Now (with #1175) this compilation passes and we get a set of requirements for firebase-admin which are guaranteed not to include msgpack. 🙌

[atugushev]

@jamescooke

Cool! Thanks for the detailed feedback! 👍

[clslgrnc]

Interesting use case @jamescooke thanks for the explanation.