Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"ERROR: No matching distribution found" in updating Pipfile.lock in installing a package from private Package Repository of GitLab.com #5021

Closed
masato-yasuda opened this issue Mar 29, 2022 · 12 comments · Fixed by #5029

Comments

@masato-yasuda
Copy link

When we try to install a private package from our Package Repository of GitLab.com, we got the following error.
(It worked in v2022.1.18. However, we got this error in v2022.3.23 and v2022.3.24.)

$ pipenv install mypackage20220329 -v --extra-index-url https://__token__:{my_secret_token}@gitlab.com/api/v4/projects/{my_project_no}/packages/pypi/simple
Installing mypackage20220329...
Installing package: mypackage20220329
Writing supplied requirement line to temporary file: 'mypackage20220329'
Installing 'mypackage20220329'
⠙ Installing mypackage20220329...$ /home/yasuda/.local/share/virtualenvs/20220329_test-EbHYGP4V/bin/python -m pip install --verbose --upgrade --exists-action=i -r /tmp/pipenv-u1u818u1-requirements/pipenv-ysi1um9s-requirement.txt -i https://__token__:{my_secret_token}@gitlab.com/api/v4/projects/{my_project_no}/packages/pypi/simple --extra-index-url https://pypi.org/simple
Using source directory: '/home/yasuda/.local/share/virtualenvs/20220329_test-EbHYGP4V/src'
Adding mypackage20220329 to Pipfile's [packages]...
✔ Installation Succeeded
Pipfile.lock (db4242) out of date, updating to (6b313a)...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
Reporter.starting()
INFO:pipenv.patched.notpip._internal.resolution.resolvelib.reporter:Reporter.starting()
Reporter.adding_requirement(SpecifierRequirement('mypackage20220329'), None)
INFO:pipenv.patched.notpip._internal.resolution.resolvelib.reporter:Reporter.adding_requirement(SpecifierRequirement('mypackage20220329'), None)
CRITICAL:pipenv.patched.notpip._internal.resolution.resolvelib.factory:Could not find a version that satisfies the requirement mypackage20220329 (from versions: none)
Traceback (most recent call last):
  File "/home/yasuda/.local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/resolvelib/resolvers.py", line 341, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "/home/yasuda/.local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/resolvelib/resolvers.py", line 173, in _add_to_criteria
    raise RequirementsConflicted(criterion)
pipenv.patched.notpip._vendor.resolvelib.resolvers.RequirementsConflicted: Requirements conflict: SpecifierRequirement('mypackage20220329')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/yasuda/.local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/resolution/resolvelib/resolver.py", line 94, in resolve
    result = self._result = resolver.resolve(
  File "/home/yasuda/.local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/resolvelib/resolvers.py", line 472, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/home/yasuda/.local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/resolvelib/resolvers.py", line 343, in resolve
    raise ResolutionImpossible(e.criterion.information)
pipenv.patched.notpip._vendor.resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=SpecifierRequirement('mypackage20220329'), parent=None)]
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/home/yasuda/.local/lib/python3.8/site-packages/pipenv/utils.py", line 903, in resolve
    results = resolver.resolve(self.constraints, check_supported_wheels=False)
  File "/home/yasuda/.local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/resolution/resolvelib/resolver.py", line 103, in resolve
    raise error from e
pipenv.patched.notpip._internal.exceptions.DistributionNotFound: No matching distribution found for mypackage20220329
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/yasuda/.local/lib/python3.8/site-packages/pipenv/resolver.py", line 766, in <module>
    main()
  File "/home/yasuda/.local/lib/python3.8/site-packages/pipenv/resolver.py", line 760, in main
    _main(parsed.pre, parsed.clear, parsed.verbose, parsed.system, parsed.write,
  File "/home/yasuda/.local/lib/python3.8/site-packages/pipenv/resolver.py", line 743, in _main
    resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages, dev)
  File "/home/yasuda/.local/lib/python3.8/site-packages/pipenv/resolver.py", line 704, in resolve_packages
    results, resolver = resolve(
  File "/home/yasuda/.local/lib/python3.8/site-packages/pipenv/resolver.py", line 685, in resolve
    return resolve_deps(
  File "/home/yasuda/.local/lib/python3.8/site-packages/pipenv/utils.py", line 1398, in resolve_deps
    results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
  File "/home/yasuda/.local/lib/python3.8/site-packages/pipenv/utils.py", line 1127, in actually_resolve_deps
    resolver.resolve()
  File "/home/yasuda/.local/lib/python3.8/site-packages/pipenv/utils.py", line 905, in resolve
    raise ResolutionFailure(message=str(e))
pipenv.exceptions.ResolutionFailure: ERROR: No matching distribution found for mypackage20220329
✘ Locking Failed!

My Pipfile is the following.

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://__token__:{my_secret_token}@gitlab.com/api/v4/projects/{my_project_no}/packages/pypi/simple"
verify_ssl = true
name = "__token__:{my_secret_token}@gitlab"

[packages]
mypackage20220329 = "*"

[dev-packages]

[requires]

My Pipfile.lock is the following.

{
    "_meta": {
        "hash": {
            "sha256": "7f7606f08e0544d8d012ef4d097dabdd6df6843a28793eb6551245d4b2db4242"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.8"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {},
    "develop": {}
}

Our package "mypackage20220329" is uploaded at GitLab.com.
We prepared the package by following the instruction at https://docs.gitlab.com/ee/user/packages/pypi_repository/ .

Note: We can't reproduce the error if the package name is "mypypipackage". It's because "mypypipackage" is already on pypi.
Therefore, we suspect that this error is caused by search at the wrong location in checking the existence of "mypackage20220329".

  • Expected location: https://token:{my_secret_token}@gitlab.com/api/v4/projects/{my_project_no}/packages/pypi/simple"
  • Current location(Wrong): https://pypi.org/simple
@Arcitec
Copy link

Arcitec commented Mar 29, 2022

I have narrowed down the bug: It happened sometime after the "2022.1.8" version of Pipenv.

You can get a working, older version of Pipenv like this:

pip install --user pipenv==2022.1.8

@Arcitec
Copy link

Arcitec commented Mar 29, 2022

There's a long ticket where we discovered the same bug here:

#4961 (comment)

PyTorch can't be installed from a PEP-503 repo anymore with the latest Pipenv. The 2022.1.8 vversion works. I have not checked all of the March 2022 releases but I know for a fact that Pipenv 2022.3.28 is broken.

@matteius
Copy link
Member

I am wondering if you are falling into an edge case of the index restriction fixes. #4983

Could you try naming the non pypi index and calling out the name on the line that requires it? For @masato-yasuda that would look like:

url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://__token__:{my_secret_token}@gitlab.com/api/v4/projects/{my_project_no}/packages/pypi/simple"
verify_ssl = true
name = "gitlab"

[packages]
mypackage20220329 = "{version="*", index="gitlab"}"

[dev-packages]

[requires]

@Arcitec
Copy link

Arcitec commented Mar 29, 2022

@matteius Confirmed. That is the problem. Results are over at:

#5022 (comment)

@masato-yasuda
Copy link
Author

@matteius
I also confirmed that Pipfile.lock is updated withour an error after changing the Pipefile like the following as you mentioned.
Pipefile

$ cat Pipfile
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://__token__:{my_secret_token}@gitlab.com/api/v4/projects/{my_project_no}/packages/pypi/simple"
verify_ssl = true
name = "gitlab"

[packages]
mypackage20220329 = {version="*", index="gitlab"}

[dev-packages]

[requires]
python_version = "3.8"

Result (No Error)

$ pipenv install mypackage20220329 -v --extra-index-url https://__token__:{my_secret_token}@gitlab.com/api/v4/projects/{my_project_no}/packages/pypi/simple
Installing mypackage20220329...
Installing package: mypackage20220329
Writing supplied requirement line to temporary file: 'mypackage20220329'
Installing 'mypackage20220329'
.$ /home/yasuda/.local/share/virtualenvs/20220329_test-EbHYGP4V/bin/python -m pip install --verbose --upgrade --exists-action=i -r /tmp/pipenv-546fve9q-requirements/pipenv-sa4asgn8-requirement.txt -i https://__token__:{my_secret_token}@gitlab.com/api/v4/projects/{my_project_no}/packages/pypi/simple --extra-index-url https://pypi.org/simple
Using source directory: '/home/yasuda/.local/share/virtualenvs/20220329_test-EbHYGP4V/src'
Adding mypackage20220329 to Pipfile's [packages]...
✔ Installation Succeeded
Installing dependencies from Pipfile.lock (57f4cc)...
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/0 — 00:00:00
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

@Bananaman
Thanks for your follow-up :)

@matteius
Copy link
Member

I have spent some time debugging this tonight. Here is what I have found;

1.) It works to specify on the CLI by index such as pipenv install --index https://download.pytorch.org/whl/cu113/ "torch==1.11.0+cu113"

2.) The problem with passing in ---extra-index-url is that it basically assumes you mean that the default --index is pypi.

3.) Pip itself says about --extra-index-url that it is used to "Search an additional index during install, in addition to PyPI"

Using this option to search for packages which are not in the main repository (such as private packages) is unsafe, per a security vulnerability called dependency confusion: an attacker can claim the package on the public repository in a way that will ensure it gets chosen over the private package.
image

4.) Hash generation was broken with package restrictions in place for some scenarios and so I have a new PR out to fi that.

In conclusion tonight, my current analysis is that we should perhaps remove the --extra-index-url argument from pipenv, as suggested by @Bananaman in this commnet #5022 (comment)
but I will indeed need to consider this more. I think though if we remove that --extra-index-url we do not need to add a new flag such --use-index-url or --set-index-url though because --index already serves the purpose of specifying the exact index to use for the package when installing via the CLI.

If you have thoughts further on this issue would love to hear them because we will want to make a decision on this for the April release.

@masato-yasuda
Copy link
Author

@matteius
Thanks for your investigation! Does it mean that a private package which has a dependent package on pypi doesn't work?
(i.e. pipenv will only see one index url. If a private package depends on a package on pypi, the dependent package cannot be installed because pipenv see only see private index which is not a pypi..)

@matteius
Copy link
Member

Does it mean that a private package which has a dependent package on pypi doesn't work?
(i.e. pipenv will only see one index url. If a private package depends on a package on pypi, the dependent package cannot be installed because pipenv see only see private index which is not a pypi..)

@masato-yasuda No this seems to work fine, since in the example of torch repositories, the are in a private repository and pull in other dependencies from pypi. Here is a sample Pipfile.lock I generated:

{
    "_meta": {
        "hash": {
            "sha256": "b5ad2e9ab98f4d253629a73ed721b21a23453d4ddc77b2df9fd52d607cc03946"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.10"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            },
            {
                "name": "downloadpytorch",
                "url": "https://download.pytorch.org/whl/",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "certifi": {
            "hashes": [
                "sha256:78884e7c1d4b00ce3cea67b44566851c4343c120abd683433ce934a68ea58872",
                "sha256:d62a0163eb4c2344ac042ab2bdf75399a71a2d8c7d47eac2e2ee91b9d6339569"
            ],
            "version": "==2021.10.8"
        },
        "charset-normalizer": {
            "hashes": [
                "sha256:2857e29ff0d34db842cd7ca3230549d1a697f96ee6d3fb071cfa6c7393832597",
                "sha256:6881edbebdb17b39b4eaaa821b438bf6eddffb4468cf344f09f89def34a8b1df"
            ],
            "markers": "python_version >= '3'",
            "version": "==2.0.12"
        },
        "colorama": {
            "hashes": [
                "sha256:5941b2b48a20143d2267e95b1c2a7603ce057ee39fd88e7329b0c292aa16869b",
                "sha256:9f47eda37229f68eee03b24b9748937c7dc3868f906e8ba69fbcbdd3bc5dc3e2"
            ],
            "markers": "platform_system == 'Windows'",
            "version": "==0.4.4"
        },
        "idna": {
            "hashes": [
                "sha256:84d9dd047ffa80596e0f246e2eab0b391788b0503584e8945f2368256d2735ff",
                "sha256:9d643ff0a55b762d5cdb124b8eaa99c66322e2157b69160bc32796e824360e6d"
            ],
            "markers": "python_version >= '3'",
            "version": "==3.3"
        },
        "numpy": {
            "hashes": [
                "sha256:07a8c89a04997625236c5ecb7afe35a02af3896c8aa01890a849913a2309c676",
                "sha256:08d9b008d0156c70dc392bb3ab3abb6e7a711383c3247b410b39962263576cd4",
                "sha256:201b4d0552831f7250a08d3b38de0d989d6f6e4658b709a02a73c524ccc6ffce",
                "sha256:2c10a93606e0b4b95c9b04b77dc349b398fdfbda382d2a39ba5a822f669a0123",
                "sha256:3ca688e1b9b95d80250bca34b11a05e389b1420d00e87a0d12dc45f131f704a1",
                "sha256:48a3aecd3b997bf452a2dedb11f4e79bc5bfd21a1d4cc760e703c31d57c84b3e",
                "sha256:568dfd16224abddafb1cbcce2ff14f522abe037268514dd7e42c6776a1c3f8e5",
                "sha256:5bfb1bb598e8229c2d5d48db1860bcf4311337864ea3efdbe1171fb0c5da515d",
                "sha256:639b54cdf6aa4f82fe37ebf70401bbb74b8508fddcf4797f9fe59615b8c5813a",
                "sha256:8251ed96f38b47b4295b1ae51631de7ffa8260b5b087808ef09a39a9d66c97ab",
                "sha256:92bfa69cfbdf7dfc3040978ad09a48091143cffb778ec3b03fa170c494118d75",
                "sha256:97098b95aa4e418529099c26558eeb8486e66bd1e53a6b606d684d0c3616b168",
                "sha256:a3bae1a2ed00e90b3ba5f7bd0a7c7999b55d609e0c54ceb2b076a25e345fa9f4",
                "sha256:c34ea7e9d13a70bf2ab64a2532fe149a9aced424cd05a2c4ba662fd989e3e45f",
                "sha256:dbc7601a3b7472d559dc7b933b18b4b66f9aa7452c120e87dfb33d02008c8a18",
                "sha256:e7927a589df200c5e23c57970bafbd0cd322459aa7b1ff73b7c2e84d6e3eae62",
                "sha256:f8c1f39caad2c896bc0018f699882b345b2a63708008be29b1f355ebf6f933fe",
                "sha256:f950f8845b480cffe522913d35567e29dd381b0dc7e4ce6a4a9f9156417d2430",
                "sha256:fade0d4f4d292b6f39951b6836d7a3c7ef5b2347f3c420cd9820a1d90d794802",
                "sha256:fdf3c08bce27132395d3c3ba1503cac12e17282358cb4bddc25cc46b0aca07aa"
            ],
            "markers": "python_version >= '3.8'",
            "version": "==1.22.3"
        },
        "requests": {
            "hashes": [
                "sha256:68d7c56fd5a8999887728ef304a6d12edc7be74f1cfa47714fc8b414525c9a61",
                "sha256:f22fa1e554c9ddfd16e6e41ac79759e17be9e492b3587efa038054674760e72d"
            ],
            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4, 3.5'",
            "version": "==2.27.1"
        },
        "torch": {
            "hashes": [
                "sha256:6c66502d4e30464abd8ede9b00ef85ac7eaf569bdf53663375a0ed3f49c4f1e5"
            ],
            "version": "==1.11.0+cu115"
        },
        "torchtext": {
            "hashes": [
                "sha256:09a047a90615febb5280bb39f69e89de515212d8d0f75b85e5ad474221f1744a",
                "sha256:174bc6b33d77d9eebbae473f98389183a0c269c68416372ca614b12cfa326969",
                "sha256:1845c480fe47c641816d770efac3a32fb927673d46b00c062a2ec377bc312cdc",
                "sha256:1db04814d0289429bebd776707ea58d251166f56cc8298956101534a20f0a5cd",
                "sha256:21b6b5a00000ad9dfa09d984a1a7b7cdd79c8310c2a780e672cdf95a2bc26c1b",
                "sha256:24b34a1b857aeef8d0c6e7b3ed2d9230d0a36df5bdb9dd5262b018675b739306",
                "sha256:2ea13e79b02d226adb122ecc8f9648c768e0b683e32fd8b3c0c0c22e8661ed6f",
                "sha256:2ecd9a32064ea5caf78c7e4843719ecabe100066b32df8d228a6f2ba6bc5ea8b",
                "sha256:365e5f640e29d2fc924cc274f024ee02e49e680b5a6d10d4d7ccdd1665b73120",
                "sha256:36a8c9d6ddbfb70904f15caade91f8deb28b35c2693347853f1bea7f65f00b40",
                "sha256:391dd25cc426ade637da7297e6d72eaf5efc5fa13a4e25cf2fa5d93119bb176b",
                "sha256:59912472689e5734ccdd134a0352e46bb36cecd3d59b323e097474f4c4e4bf9a",
                "sha256:63d9b6e94af6092c472c77e26cf466599eccc1e3a0facc59c87d7611b5afcda6",
                "sha256:66792b1e6d3929944a8d47ad371c71de07a4023f5692385ba7e50c61e111f2f2",
                "sha256:87530abac557e9702d41edd89c94ccbc4871d43c0f479af97ae3342836e875ec",
                "sha256:8bbf5649e4dc691535920437c09017fcab7cfed7125ed0cfe1e1a3bbe792ab93",
                "sha256:af04961985c419579662e703243fa592d74fc44e2863a44c23214b377cf426d8",
                "sha256:b7fa7557fe5542b58c09abb4bf14e0c6ed6afe653606e318080ba0718d3e1eda",
                "sha256:bce2a8f6f3a82974b950ee526640b3f747e7bade32b92d0747687bfe76af8d81"
            ],
            "index": "downloadpytorch",
            "version": "==0.12.0"
        },
        "tqdm": {
            "hashes": [
                "sha256:4230a49119a416c88cc47d0d2d32d5d90f1a282d5e497d49801950704e49863d",
                "sha256:6461b009d6792008d0000e1b0c7ca50195ec78c0e808a3a6b668a56a3236c3a5"
            ],
            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
            "version": "==4.63.1"
        },
        "typing-extensions": {
            "hashes": [
                "sha256:1a9462dcc3347a79b1f1c0271fbe79e844580bb598bafa1ed208b94da3cdcd42",
                "sha256:21c85e0fe4b9a155d0799430b0ad741cdce7e359660ccbd8b530613e8df88ce2"
            ],
            "markers": "python_version >= '3.6'",
            "version": "==4.1.1"
        },
        "urllib3": {
            "hashes": [
                "sha256:44ece4d53fb1706f667c9bd1c648f5469a2ec925fcf3a776667042d645472c14",
                "sha256:aabaf16477806a5e1dd19aa41f8c2b7950dd3c746362d7e3223dbe6de6ac448e"
            ],
            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4' and python_version < '4'",
            "version": "==1.26.9"
        }
    },
    "develop": {}
}

@masato-yasuda
Copy link
Author

@matteius
Thanks for the information! As long as it works fine, I think we are OK :)

@juanitosvq
Copy link

juanitosvq commented May 10, 2022

Hi everyone,

@matteius we seem to be having a similar issue. In our case, one of our private packages has a dependent package also in our private repo.

It works fine with v2022.1.8, but failing with v2022.5.2.

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://aws:$CODEARTIFACT_AUTH_TOKEN@evertz-io-############.d.codeartifact.us-east-1.amazonaws.com/pypi/my-repo/simple/"
verify_ssl = true
name = "repo.io"

[packages]
identity-lib = {version = "==2.6.1",index = "repo.io"}

[requires]
python_version = "3.7"

[pipenv]
allow_prereleases = true

This fails to find a matching distribution for one of the dependencies of identity-lib, which happens to also be part of repo.io. Any ideas? I will be happy to open a new issue altogether if you think that may be better.

Thanks in advance

@matteius
Copy link
Member

@juanitosvq You have at least a couple options without any further pipenv development to get this scenario working, though perhaps you could propose imporvements to the docs as well. I've added some improvements already to https://pipenv.pypa.io/en/latest/advanced/#specifying-package-indexes

1.) You can try swapping the order of your source definitions so that "repo.io" comes first. This makes that one the default index for not specified packages. This may require that you have mirrored your pypi packages to the private server, while best practice, may not always be possible.
2.) Alternatively, you would add the sub-dependencies to the your packages list with {version = "*",index = "repo.io"} and that will allow the combination of the sub-version and the index specification to work in tandem.

I hope this helps!

@juanitosvq
Copy link

Hey @matteius thanks for the quick reply! I'll reply in #5053 so that we don't have multiple issues talking about the same thing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants