Multiple sources on same domain

[MyrikLD]

Issue description

ResolutionFailure on install packages from multiple sources with equal domains

Expected result

Package installing

Actual result

Locking [dev-packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success!
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✘ Locking Failed!
[ResolutionFailure]: File "/usr/lib/python3.9/site-packages/pipenv/resolver.py", line 741, in _main
[ResolutionFailure]: resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages, dev)
[ResolutionFailure]: File "/usr/lib/python3.9/site-packages/pipenv/resolver.py", line 702, in resolve_packages
[ResolutionFailure]: results, resolver = resolve(
[ResolutionFailure]: File "/usr/lib/python3.9/site-packages/pipenv/resolver.py", line 684, in resolve
[ResolutionFailure]: return resolve_deps(
[ResolutionFailure]: File "/usr/lib/python3.9/site-packages/pipenv/utils.py", line 1395, in resolve_deps
[ResolutionFailure]: results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
[ResolutionFailure]: File "/usr/lib/python3.9/site-packages/pipenv/utils.py", line 1108, in actually_resolve_deps
[ResolutionFailure]: resolver.resolve()
[ResolutionFailure]: File "/usr/lib/python3.9/site-packages/pipenv/utils.py", line 833, in resolve
[ResolutionFailure]: raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
First try clearing your dependency cache with $ pipenv lock --clear, then try the original command again.
Alternatively, you can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: Could not find a version that matches package_two (from -r /tmp/pipenvamergd9lrequirements/pipenv-djbch16n-constraints.txt (line 14))
No versions found
Were https://pypi.org/simple or https://__token__:***@gitlab.com/api/v4/projects/123/packages/pypi/simple or https://__token__:***@gitlab.com/api/v4/projects/456/packages/pypi/simple reachable?

Steps to replicate

pipenv lock

---

$ pipenv --support

Pipenv version: '2020.11.15'

Pipenv location: '~/.local/share/virtualenvs/scraper-mWBbuJI4/lib/python3.9/site-packages/pipenv'

Python location: '~/.local/share/virtualenvs/scraper-mWBbuJI4/bin/python'

Python installations found:

• 3.9.2: ~/.local/share/virtualenvs/scraper-mWBbuJI4/bin/python3
• 3.9.2: ~/.local/share/virtualenvs/scraper-mWBbuJI4/bin/python3.9
• 3.9.2: ~/.local/share/virtualenvs/scraper-mWBbuJI4/bin/python3
• 3.9.2: ~/.local/share/virtualenvs/scraper-mWBbuJI4/bin/python3.9
• 3.9.2: /usr/bin/python3.9
• 3.9.2: /usr/bin/python3
• 3.8.8: /usr/bin/python3.8
• 2.7.18: /usr/bin/python2.7
• 2.7.18: /usr/bin/python2

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.9.2',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '5.11.10-1-MANJARO',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP PREEMPT Thu Mar 25 17:14:55 UTC 2021',
 'python_full_version': '3.9.2',
 'python_version': '3.9',
 'sys_platform': 'linux'}

System environment variables:

• HOME
• DBUS_SESSION_BUS_ADDRESS
• XDG_CURRENT_DESKTOP
• LC_MONETARY
• MONGO_USERNAME
• PATH
• PIP_PYTHON_PATH
• REDIS_PORT
• SHELL
• XDG_MENU_PREFIX
• PIPENV_ACTIVE
• LC_PAPER
• SSH_AGENT_PID
• QT_AUTO_SCREEN_SCALE_FACTOR
• XDG_VTNR
• USER
• PIP_DISABLE_PIP_VERSION_CHECK
• LC_TIME
• XDG_SESSION_TYPE
• LC_MEASUREMENT
• XDG_SEAT
• DESKTOP_SESSION
• GTK_MODULES
• LC_IDENTIFICATION
• PYTHONDONTWRITEBYTECODE
• MOTD_SHOWN
• LC_NAME
• LC_NUMERIC
• TERM
• GDK_BACKEND
• XDG_SESSION_CLASS
• GTK3_MODULES
• XDG_CONFIG_DIRS
• PWD
• __pipenv_fish_initial_pwd
• SESSION_MANAGER
• XDG_SESSION_ID
• EDITOR
• LOGNAME
• MONGO_PASSWORD
• QT_QPA_PLATFORMTHEME
• XDG_RUNTIME_DIR
• REDIS_HOST
• VTE_VERSION
• LC_ADDRESS
• LC_TELEPHONE
• XAUTHORITY
• MKLROOT
• XDG_DATA_DIRS
• JOURNAL_STREAM
• DISPLAY
• MAIL
• LANG
• SSH_AUTH_SOCK
• SHLVL
• COLORTERM
• LS_COLORS
• INVOCATION_ID
• MONGO_HOST
• GTK2_RC_FILES
• VIRTUAL_ENV
• PIP_SHIMS_BASE_MODULE
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

• PIPENV_ACTIVE: 1

Debug–specific environment variables:

• PATH: /home/linuxbrew/.linuxbrew/opt/[email protected]/bin:package_two/.local/share/virtualenvs/scraper-mWBbuJI4/bin:package_two/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/var/lib/flatpak/exports/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl
• SHELL: /bin/fish
• EDITOR: /usr/bin/nano
• LANG: en_US.UTF-8
• PWD: /work/upsilon/liveart/scraper/laa_scrapy
• VIRTUAL_ENV: package_two/.local/share/virtualenvs/scraper-mWBbuJI4

---

Contents of Pipfile ('/work/upsilon/liveart/scraper/Pipfile'):

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[[source]]
url = "https://__token__:***@gitlab.com/api/v4/projects/123/packages/pypi/simple"
verify_ssl = true
name = "scraper"

[[source]]
url = "https://__token__:***@gitlab.com/api/v4/projects/456/packages/pypi/simple"
verify_ssl = true
name = "api"

[packages]
package_one = {version="*", index="api"}
package_two = {version="*", index="scraper"}

[requires]
python_version = "3"

[pipenv]
allow_prereleases = true

[matteius]

@MyrikLD Since this involves private Pypis can you re-test with the latest version 2022.1.8?

[matteius]

I suspect the newest version of 2022.4.8 would not have this issue.

[ThomasSteinbach]

I am facing the same problem. Also with version 2022.4.8. The last working version for me is 2021.11.23.

It seems like the new version generally does not handle multiple sources on locking

[[source]]
url = "https://repo.example.com/artifactory/api/pypi/default-pypi-3rdparty/simple"
verify_ssl = true
name = "default-pypi-3rdparty"

[[source]]
url = "https://repo.example.com/artifactory/api/pypi/test-cdk-pypi-dev-local/simple"
verify_ssl = true
name = "test-cdk-pypi-dev-local"

[[source]]
url = "https://repo.example.com/artifactory/api/pypi/test-cdk-pypi-prd-local/simple"
verify_ssl = true
name = "test-cdk-pypi-prd-local"
...

Maybe pipenv only checks the first source, which leads to:

sarezdbmac:test-cdk thomass$ pipenv install -v
Pipfile.lock (08c22b) out of date, updating to (36c622)...
Locking [dev-packages] dependencies...
Building requirements...
Resolving dependencies...
Reporter.starting()
INFO:pipenv.patched.notpip._internal.resolution.resolvelib.reporter:Reporter.starting()
Reporter.adding_requirement(SpecifierRequirement('flake8'), None)
INFO:pipenv.patched.notpip._internal.resolution.resolvelib.reporter:Reporter.adding_requirement(SpecifierRequirement('flake8'), None)
Reporter.adding_requirement(SpecifierRequirement('test-gcip~=2.0'), None)
INFO:pipenv.patched.notpip._internal.resolution.resolvelib.reporter:Reporter.adding_requirement(SpecifierRequirement('test-gcip~=2.0'), None)
CRITICAL:pipenv.patched.notpip._internal.resolution.resolvelib.factory:Could not find a version that satisfies the requirement test-gcip~=2.0 (from versions: none)
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pipenv/patched/notpip/_vendor/resolvelib/resolvers.py", line 341, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pipenv/patched/notpip/_vendor/resolvelib/resolvers.py", line 173, in _add_to_criteria
    raise RequirementsConflicted(criterion)
pipenv.patched.notpip._vendor.resolvelib.resolvers.RequirementsConflicted: Requirements conflict: SpecifierRequirement('test-gcip~=2.0')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pipenv/patched/notpip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve
    collected.requirements, max_rounds=try_to_avoid_resolution_too_deep
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pipenv/patched/notpip/_vendor/resolvelib/resolvers.py", line 472, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pipenv/patched/notpip/_vendor/resolvelib/resolvers.py", line 343, in resolve
    raise ResolutionImpossible(e.criterion.information)
pipenv.patched.notpip._vendor.resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=SpecifierRequirement('test-gcip~=2.0'), parent=None)]
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 661, in resolve
    results = resolver.resolve(self.constraints, check_supported_wheels=False)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pipenv/patched/notpip/_internal/resolution/resolvelib/resolver.py", line 103, in resolve
    raise error from e
pipenv.patched.notpip._internal.exceptions.DistributionNotFound: No matching distribution found for test-gcip~=2.0
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pipenv/resolver.py", line 857, in <module>
    main()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pipenv/resolver.py", line 852, in main
    dev=parsed.dev,
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pipenv/resolver.py", line 823, in _main
    pre, clear, verbose, system, write, requirements_dir, packages, dev
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pipenv/resolver.py", line 778, in resolve_packages
    requirements_dir=requirements_dir,
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pipenv/resolver.py", line 759, in resolve
    req_dir=requirements_dir,
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 1074, in resolve_deps
    req_dir=req_dir,
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 862, in actually_resolve_deps
    resolver.resolve()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 663, in resolve
    raise ResolutionFailure(message=str(e))
pipenv.exceptions.ResolutionFailure: ERROR: No matching distribution found for test-gcip~=2.0
✘ Locking Failed! 

[matteius]

@ThomasSteinbach what is your example Pipfile look like?

[ThomasSteinbach]

@matteius here you are. I only replaced some names due to confidentiality:

[[source]]
url = "https://repo.example.com/artifactory/api/pypi/default-pypi-3rdparty/simple"
verify_ssl = true
name = "default-pypi-3rdparty"

[[source]]
url = "https://repo.example.com/artifactory/api/pypi/myteam-cdk-pypi-dev/simple"
verify_ssl = true
name = "myteam-cdk-pypi-dev"

[[source]]
url = "https://repo.example.com/artifactory/api/pypi/myteam-cdk-pypi-prd/simple"
verify_ssl = true
name = "myteam-cdk-pypi-prd"

[[source]]
url = "https://repo.example.com/artifactory/api/pypi/myteam-gcip-pypi-dev/simple"
verify_ssl = true
name = "myteam-gcip-pypi-dev"

[[source]]
url = "https://repo.example.com/artifactory/api/pypi/myteam-gcip-pypi-prd/simple"
verify_ssl = true
name = "myteam-gcip-pypi-prd"

[packages]
# Minor versions in CDK might not be a stable change.
# That is why we only auto update micro version.
aws-cdk-lib = "~=2.18.0"
setuptools-git-versioning = "~=1.8"
boto3 = "~=1.20"
pyaml = ">=21"
toml = "~=0.10"
jinja2 = "~=3.0"

[dev-packages]
myteam-gcip = "~=2.0"
black = "*"
isort = "*"
flake8 = "*"
mypy = "*"
pytest-cov = "*"
pytest-mock = "*"
tomli = "~=2.0"

[requires]
python_version = "3.7"

All packages come from the first source (a proxy repository of pypi). Only the myteam-gcip package comes from the source named myteam-gcip-pypi-prd. The other sources are not used in this project. Those sources are just the default list of sources in our team.

[matteius]

@ThomasSteinbach Your problem is different from the original issue report. You will need to pin your package to indexes whereby the first index in your Pipfile is the default that all unspecified packages will use. This first would have been an issue for you in 2022.3.23 https://pipenv.pypa.io/en/latest/changelog/#id20 due to

Patched our vendored Pip to fix: Pipenv Lock (Or Install) Does Not Respect Index Specified For A Package. #4637

For more context see updated documentation on indexes: https://pipenv.pypa.io/en/latest/advanced/#specifying-package-indexes

See also:

• "ERROR: No matching distribution found" in updating Pipfile.lock in installing a package from private Package Repository of GitLab.com #5021
• ERROR: No matching distribution found for torch==1.11.0+cu113 when using --extra-index-url #5022
• hash generation of index restricted package missing from the the Pipfile.lock #5023

[matteius]

@ThomasSteinbach So in your case update to use:

myteam-gcip = {version= "~=2.0", index="myteam-gcip-pypi-prd"}

[ThomasSteinbach]

@MyrikLD @matteius I am absolutely sorry for seizing this issue for my problem.

Thank you @matteius for the explanation.

[matteius]

@ThomasSteinbach No worries mate, I'm glad yours was an easy fix. Also there is an open issue/PR about giving a non-default option to still install from all sources as before for those that need it, but if you pin this way you'll get optimized installation as well as it will only search the indexes you expect to get the package from.

[matteius]

@MyrikLD If you determine this is still an issue with the latest versions of pipenv and would like more support, just comment back with additional details. Until then I am closing this issue.

[MyrikLD]

Thank you, fixed in 2022.4.8