pip 20.2 incompatibility

[Nicusor97]

pip-compile fails when trying to use the latest version of pip in the finding dependencies process

Environment Versions

1. OS Type => Windows
2. Python version: $ python -V => Python 2.7.8 and Python 3.6
3. pip version: $ pip --version => 20.2
4. pip-tools version: $ pip-compile --version => 5.3.0

Steps to replicate

1. Create a requirements file (with modules that have a lot of dependencies - as in my side)
2. Run pip-compile on this file : pip-compile -v --upgrade --no-header -r dev_requirements.txt -o frozen_requirements.txt

Expected result

pip-compile completes successfully

Actual result

The pip-compile process fails in the finding dependencies process after 10 rounds.

Here is my call stack:
Result of round 10: not stable Traceback (most recent call last): File "C:\Python36\Lib\runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "C:\Python36\Lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "E:\build_infra_repos\package_checker\pip_compile_venv3\Scripts\pip-compile.exe\__main__.py", line 9, in <module> File "e:\build_infra_repos\package_checker\pip_compile_venv3\lib\site-packages\click\core.py", line 829, in __call__ return self.main(*args, **kwargs) File "e:\build_infra_repos\package_checker\pip_compile_venv3\lib\site-packages\click\core.py", line 782, in main rv = self.invoke(ctx) File "e:\build_infra_repos\package_checker\pip_compile_venv3\lib\site-packages\click\core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "e:\build_infra_repos\package_checker\pip_compile_venv3\lib\site-packages\click\core.py", line 610, in invoke return callback(*args, **kwargs) File "e:\build_infra_repos\package_checker\pip_compile_venv3\lib\site-packages\click\decorators.py", line 21, in new_func return f(get_current_context(), *args, **kwargs) File "e:\build_infra_repos\package_checker\pip_compile_venv3\lib\site-packages\piptools\scripts\compile.py", line 458, in cli results = resolver.resolve(max_rounds=max_rounds) File "e:\build_infra_repos\package_checker\pip_compile_venv3\lib\site-packages\piptools\resolver.py", line 164, in resolve "This is likely a bug.".format(max_rounds=max_rounds) RuntimeError: No stable configuration of concrete packages could be found for the given constraints after 10 rounds of resolving. This is likely a bug.

I checked the code and the default version for the max_rounds is 10 but not sure how the process gets to do 10 rounds. With the previous version of pip and the same version of pip-tools, the process of finding the dependencies process is ok in only 5 rounds.

[atugushev]

Hello @Nicusor97,

Thanks for the report! Have you tried to increase max rounds using --max-rounds option?

[Nicusor97]

Hi @atugushev ,

I already tried to use the --max-rounds option and it worked. This will fix my pipeline but if one of our clients will use the combination of the latest pip and pip-tools on one of their requirements.txt (that may be huge) what will happen? I mean, on my side, using the latest pip and pip-tools, my requirements are resolved in 12 rounds after I increased the max rounds to 20 instead of 10(which is by default ) (with the previous pip version the requirements were resolved in 6 rounds)

[atugushev]

@Nicusor97

Perhaps, we should increase the default --max-round number in pip-tools (20 is okay). Feel free to send a PR.

[Nicusor97]

@atugushev

is this an intended change? I mean, maybe there is a bug in the process and if we will increase the number of the --max-rounds we'll only somehow skip this bug ... If the latest version of pip in combination with the latest version of pip-tools is supposed to increase the number of rounds then the change that we are talking is normal but are we sure about this ?

[atugushev]

@Nicusor97

Nothing has changed on the pip-tools's resolver side. Need to research what's changed in pip that caused this. In the meantime, nothing wrong with increasing the default value, because 10 is too small, for example in the pip's new resolver it was 100 at first, but later has changed to 100000+ (IIRC).

[Nicusor97]

@atugushev

Ok, I agree. I can do the changes needed to increase the number of the default value for the --max-rounds from 10 to 20, I guess. Or should we change it to 100? (as the first value of the pip's new resolver) . Or pip-tools should default to "the number of rounds used by the version of pip being used" ?
Let me know your opinion

[atugushev]

@Nicusor97

Let's increase it to 20 and see how it goes. Could you attach your requirements.in, that would help a lot to investigate the issue?

[vphilippon]

Hello! Joining in with @Nicusor97 (we are colleagues working on the same project).

The issue occurs with our internal package(s) on a private index, so sharing the requirements.in won't allows to reproduce.

One important thing to add that we figured out: the number of round to isn't constant anymore. Sometime it takes 10, sometimes 12.
This is a red flag I'm looking into right now. The pip-tools resolver algo should be constant in its number of rounds: It always as been for as far as I've been aware of it. And if we think about it, it really should be constant.

FYI, I once met an issue with pipenv in the past when it integrated a patched version of pip-tools for its dependencies resolving where the number of rounds was unstable, and it was related to a change done on the requirements objects returned by pip (in get_best_match IIRC). I might have to go digging trying to find what exactly caused this.

In any case, that unstable number of rounds points to a new bug IMO, so increasing the number of rounds will really just be a workaround until we find why the number of rounds taken isn't stable.

I'll keep digging and come back with my findings 👷 🔍

[lorenabalan]

We have come across this on our CI for Kedro as well. requirements.in includes kedro requirements in addition to:

black==v19.10b0
flake8>=3.7.9, <4.0
ipython~=7.0
isort>=4.3.21, <5.0
jupyter~=1.0
jupyter_client~=5.1
jupyterlab==0.31.1
nbstripout==0.3.3
pytest-cov~=2.5
pytest-mock>=1.7.1, <2.0
pytest~=5.0
wheel==0.32.2

Using pip 20.2.

[atugushev]

Hello @vphilippon, thanks for the detailed info! Yeah, unconstant rounds must be a bug.

[vphilippon]

I've found part of the reason for the unconstant number of rounds. Here's a bit of the end of the pip-compile verbose output I get with my internal project:

New dependencies found in this round:

  adding [u'omega-metrics', '<3.0,>=1.0.0,>=1.1.0,>=1.2.1,>=2.0.0', '[]']

  adding [u'omega-zookeeperdbi', '<2.0,>=1.5.0,>=1.6.0', '[]']

  adding [u'requests', '<3.0,>=2.7', "['security']"]

Removed dependencies in this round:

  removing [u'omega-metrics', '<3.0,>=1.0,>=1.1.0,>=1.2.1,>=2.0.0', '[]']

  removing [u'omega-zookeeperdbi', '<2.0,>=1.5,>=1.6.0', '[]']

  removing [u'requests', '<3.0,>=2.7.0', "['security']"]

Notice how the dependencies change are just about changes on trailling .0 and such.
The dependencies are effectively the same, and only differ in the representation.

In the complete output, I've noted that some packages depends on requests>=2.7, and others on requests>=2.7.0. It seems that we correctly "merge" those in the resolver requirements combination logic, but we don't keep the same final format each time, and the logic that checks for dependencies changes between rounds is sensitive to that.

[Jyhess]

Hi,

Same random issue in our CI.
Seems fixed with --max-round 20.

Here my configuration if it can help:

1. OS Type: 5.7.9-100.fc31.x86_64
2. Python version: $ python -V: Python 3.7.7
3. pip version: $ pip --version pip 20.2 from /usr/local/lib/python3.7/site-packages/pip (python 3.7)
4. pip-tools version: $ pip-compile --version pip-compile, version 5.3.0

Command line:
pip-compile ./*.in --output-file ./requirements.txt

 Traceback (most recent call last):
  File "/builds/.venv3/bin/pip-compile", line 10, in <module>
    sys.exit(cli())
  File "/builds/.venv3/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/builds/.venv3/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/builds/.venv3/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/builds/.venv3/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/builds/.venv3/lib/python3.7/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/builds/.venv3/lib/python3.7/site-packages/piptools/scripts/compile.py", line 458, in cli
    results = resolver.resolve(max_rounds=max_rounds)
  File "/builds/.venv3/lib/python3.7/site-packages/piptools/resolver.py", line 164, in resolve
    "This is likely a bug.".format(max_rounds=max_rounds)
RuntimeError: No stable configuration of concrete packages could be found for the given constraints after 10 rounds of resolving.
This is likely a bug.

With files:

requirements.txt

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --output-file=./requirements.txt ./requirements_codegen.in ./requirements_deployment.in ./requirements_ide.in ./requirements_misc.in ./requirements_svc.in ./requirements_tests.in ./requirements_tools.in
#
--index-url https://nexus.da.rsint.net/repository/pypi/simple

aioelasticsearch==0.6.0   # via -r ./requirements_svc.in
aiofiles==0.5.0           # via -r ./requirements_svc.in
aiohttp-jinja2==1.2.0     # via -r ./requirements_svc.in, connexion
aiohttp==3.6.2            # via -r ./requirements_svc.in, aioelasticsearch, aiohttp-jinja2, connexion, pytest-aiohttp
ansible==2.9              # via -r ./requirements_deployment.in
apipkg==1.5               # via execnet
appdirs==1.4.4            # via black, virtualenv
astroid==2.4.2            # via pylint
async-property==0.2.1     # via -r ./requirements_tools.in
async-timeout==3.0.1      # via aiohttp
attrs==19.3.0             # via aiohttp, black, hypothesis, pytest
bandit==1.6.2             # via -r ./requirements_ide.in
bcrypt==3.1.7             # via paramiko
beautifulsoup4==4.9.1     # via -r ./requirements_tools.in
black==19.10b0            # via -r ./requirements_codegen.in, -r ./requirements_ide.in
cached-property==1.5.1    # via -r ./requirements_tools.in, docker-compose
cachetools==4.1.1         # via google-auth
certifi==2020.6.20        # via requests
cffi==1.14.0              # via bcrypt, cryptography, pynacl
chardet==3.0.4            # via aiohttp, requests
click==7.1.2              # via black, clickclick, flask
clickclick==1.2.2         # via connexion
colorama==0.4.3           # via -r ./requirements_svc.in
colorlog==4.1.0           # via -r ./requirements_tools.in
compose==1.1.1            # via -r ./requirements_misc.in
connexion-2018.0.dev2-py2.py3-none-any.whl  # via -r ./requirements_svc.in
coverage==4.5.4           # via -r ./requirements_tests.in, pytest-cov
cryptography==2.8         # via -r ./requirements_svc.in, ansible, paramiko
deepdiff==5.0.1           # via -r ./requirements_tests.in
distlib==0.3.1            # via virtualenv
distro==1.5.0             # via docker-compose
dnspython==2.0.0          # via -r ./requirements_svc.in
docker-compose==1.26.2    # via -r ./requirements_tools.in
docker[ssh]==4.2.2        # via docker-compose
dockerpty==0.4.1          # via docker-compose
docopt==0.6.2             # via docker-compose
elasticsearch==7.5.1      # via -r ./requirements_svc.in, aioelasticsearch
execnet==1.7.1            # via pytest-xdist
filelock==3.0.12          # via virtualenv
flask==1.1.2              # via connexion
gitdb==4.0.5              # via gitpython
gitpython==3.1.7          # via bandit
google-api-core==1.21.0   # via opencensus
google-auth==1.19.1       # via google-api-core
googleapis-common-protos==1.52.0  # via google-api-core
grpcio==1.30.0            # via -r ./requirements_svc.in
hypothesis==4.9.0         # via -r ./requirements_tests.in
idna==2.10                # via requests, yarl
importlib-metadata==1.7.0  # via pluggy, pytest, pytest-randomly, stevedore, virtualenv
inflection==0.5.0         # via connexion
isort==4.3.21             # via pylint
itsdangerous==1.1.0       # via flask
jinja2==2.11.2            # via -r ./requirements_codegen.in, aiohttp-jinja2, ansible, flask, swagger-ui-bundle
jsonschema==2.6.0         # via connexion, docker-compose, openapi-spec-validator
lazy-object-proxy==1.4.3  # via astroid
markupsafe==1.1.1         # via jinja2
mccabe==0.6.1             # via pylint
more-itertools==8.4.0     # via pytest
motor==2.1.0              # via -r ./requirements_svc.in
multidict==4.7.6          # via aiohttp, yarl
oauthlib==3.1.0           # via -r ./requirements_svc.in
openapi-spec-validator==0.2.8  # via connexion
opencensus-context==0.1.1  # via opencensus
opencensus-ext-zipkin==0.2.2  # via -r ./requirements_svc.in
opencensus==0.7.10        # via -r ./requirements_svc.in, opencensus-ext-zipkin
ordered-set==4.0.2        # via deepdiff
packaging==20.4           # via pytest
paramiko==2.7.1           # via docker
pathspec==0.8.0           # via black
pbr==5.4.5                # via stevedore
pluggy==0.13.1            # via pytest
plumbum==1.6.9            # via -r ./requirements_svc.in, -r ./requirements_tools.in
protobuf==3.12.2          # via google-api-core, googleapis-common-protos
psutil==5.7.2             # via -r ./requirements_tools.in
py==1.9.0                 # via pytest, pytest-datafiles
pyasn1-modules==0.2.8     # via google-auth
pyasn1==0.4.8             # via pyasn1-modules, rsa
pycparser==2.20           # via cffi
pyjwt==1.7.1              # via -r ./requirements_svc.in
pylint-exit==1.2.0        # via -r ./requirements_ide.in
pylint==2.5.3             # via -r ./requirements_ide.in
pymongo==3.10.1           # via motor
pynacl==1.4.0             # via paramiko
pyparsing==2.4.2          # via -r ./requirements_svc.in, packaging
pytest-aiohttp==0.3.0     # via -r ./requirements_tests.in
pytest-cov==2.10.0        # via -r ./requirements_tests.in
pytest-datafiles==2.0     # via -r ./requirements_tests.in
pytest-forked==1.2.0      # via pytest-xdist
pytest-mock==2.0.0        # via -r ./requirements_tests.in
pytest-randomly==3.4.1    # via -r ./requirements_tests.in
pytest-xdist==1.33.0      # via -r ./requirements_tests.in
pytest==5.4.3             # via -r ./requirements_tests.in, pytest-aiohttp, pytest-cov, pytest-datafiles, pytest-forked, pytest-mock, pytest-randomly, pytest-xdist
python-dateutil==2.8.1    # via -r ./requirements_codegen.in, -r ./requirements_svc.in
python-dotenv==0.14.0     # via docker-compose
pytz==2020.1              # via google-api-core
pyyaml==5.3.1             # via -r ./requirements_tools.in, ansible, bandit, clickclick, connexion, docker-compose, openapi-spec-validator
regex==2020.7.14          # via black
requests==2.24.0          # via connexion, docker, docker-compose, google-api-core
rfc3986==1.4.0            # via -r ./requirements_codegen.in, -r ./requirements_svc.in
rsa==4.6                  # via google-auth
six==1.15.0               # via astroid, bandit, bcrypt, connexion, cryptography, docker, docker-compose, dockerpty, google-api-core, google-auth, grpcio, openapi-spec-validator, packaging, protobuf, pynacl, pytest-xdist, python-dateutil, structlog, virtualenv, websocket-client
smmap==3.0.4              # via gitdb
soupsieve==2.0.1          # via beautifulsoup4
stevedore==3.2.0          # via bandit
structlog==20.1.0         # via -r ./requirements_svc.in
swagger-ui-bundle==0.0.6  # via -r ./requirements_svc.in, connexion
texttable==1.6.2          # via docker-compose
toml==0.10.1              # via black, pylint
typed-ast==1.4.1          # via astroid, black
urllib3==1.25.9           # via elasticsearch, requests
virtualenv==20.0.27       # via -r ./requirements_tools.in
wcwidth==0.2.5            # via pytest
websocket-client==0.57.0  # via docker, docker-compose
werkzeug==0.16.0          # via -r ./requirements_svc.in, flask
wrapt==1.12.1             # via astroid
yarl==1.4.2               # via -r ./requirements_codegen.in, aiohttp
zipp==3.1.0               # via importlib-metadata

# The following packages are considered to be unsafe in a requirements file:
# setuptools

requirements_codegen.in

# codegen
yarl
rfc3986
jinja2
python-dateutil
black

requirements_deployment.in

# cluster deployment (mongodb)
ansible==2.9

requirements_ide.in

pylint
pylint-exit
bandit
black

requirements_misc.in

# For docker compose
compose

requirements_svc.in

connexion-2018.0.dev2-py2.py3-none-any.whl[aiohttp,swagger-ui]
werkzeug == 0.16.0
swagger-ui-bundle >= 0.0.2
aiohttp >= 3.5.4
aiohttp_jinja2 >= 1.1.0
aiofiles >= 0.4.0
structlog >= 18.2.0
colorama >= 0.4.0
motor >= 2.0.0
dnspython >= 1.16.0
aioelasticsearch >= 0.6.0
elasticsearch == 7.5.1
pyjwt >= 1.7.1
cryptography == 2.8
python-dateutil
oauthlib
pyparsing==2.4.2
plumbum
rfc3986
opencensus>=0.7.6
opencensus-ext-zipkin>=0.2.2
grpcio>=1.26.0

requirements_tests.in

pytest
pytest-aiohttp
pytest-cov
pytest-mock < 3.0.0
pytest-xdist
pytest-datafiles
pytest-randomly
coverage < 5.0
deepdiff
hypothesis<4.10.0

requirements_tools.in

psutil
colorlog
cached_property
virtualenv
PyYAML
async_property
beautifulsoup4

# adr_manager
plumbum

# For Stratus CLI
async_property
cached_property
beautifulsoup4
plumbum
psutil
docker-compose

Connexion wheel is built from https://github.com/Jyhess/connexion/tree/custom

[vphilippon]

I think I've found a fix, I'm testing it and adding some tests for it. A PR should come soon-ish™.

In short, I'm improving the RequirementSummary's __eq__ and __hash__ to not be sensible to variations.
FYI, packaging.specifiers.SpecifierSet's __eq__ and __hash__ returns the same value for >=1.0 and >=1.0.0, for example.

[vphilippon]

pip-tools 5.3.1 was released! 🚀