Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Specifying 'extras' with remote wheel causes exception #5536

Closed
elanou opened this issue Dec 14, 2022 · 7 comments · Fixed by #5555
Closed

Specifying 'extras' with remote wheel causes exception #5536

elanou opened this issue Dec 14, 2022 · 7 comments · Fixed by #5555
Assignees
Labels
Category: Requirement PR: awaiting-review The PR related to this issue is awaiting review by a maintainer. Status: In Progress This item is in progress.

Comments

@elanou
Copy link

elanou commented Dec 14, 2022

Issue description

Trying to install wheel from remote file with 'extras' specified in pipfile causes exception.

pipfile ex:
'spacy = {file = "https://files.pythonhosted.org/packages/8a/2a/fe2870e216511ed12b9b5acf89228abc0c76832b9572501e0b895a9e129d/spacy-3.4.3-cp39-cp39-macosx_10_9_x86_64.whl", extras = ["transformers"]}'

Expected result

Install wheel and extras

Actual result

Traceback (most recent call last):
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/patched/pip/_vendor/packaging/requirements.py", line 102, in init
req = REQUIREMENT.parseString(requirement_string)
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/patched/pip/_vendor/pyparsing/core.py", line 1141, in parse_string
raise exc.with_traceback(None)
pipenv.patched.pip._vendor.pyparsing.exceptions.ParseException: Expected string_end, found '#' (at char 12), (line:1, col:13)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/patched/pip/_vendor/pkg_resources/init.py", line 3101, in init
super(Requirement, self).init(requirement_string)
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/patched/pip/_vendor/packaging/requirements.py", line 104, in init
raise InvalidRequirement(
pipenv.patched.pip._vendor.packaging.requirements.InvalidRequirement: Parse error at "'#egg='": Expected string_end
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 275, in parse_line
req = Requirement.from_line(line)
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 2684, in from_line
r = file_req_from_parsed_line(parsed_line)
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 3084, in file_req_from_parsed_line
"req": parsed_line.requirement,
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 475, in requirement
self.parse_requirement()
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 1081, in parse_requirement
self._requirement = init_requirement(name) # type: PackagingRequirement
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/vendor/requirementslib/models/utils.py", line 192, in init_requirement
req = Requirement.parse(name)
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/patched/pip/_vendor/pkg_resources/init.py", line 3148, in parse
req, = parse_requirements(s)
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/patched/pip/_vendor/pkg_resources/init.py", line 3094, in parse_requirements
yield Requirement(line)
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/patched/pip/_vendor/pkg_resources/init.py", line 3103, in init
raise RequirementParseError(str(e))
pipenv.patched.pip._vendor.pkg_resources.RequirementParseError: Parse error at "'#egg='": Expected string_end
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/resolver.py", line 816, in
main()
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/resolver.py", line 802, in main
_main(
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/resolver.py", line 782, in _main
resolve_packages(
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/resolver.py", line 730, in resolve_packages
results, resolver = resolve(
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/resolver.py", line 709, in resolve
return resolve_deps(
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 1105, in resolve_deps
results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 891, in actually_resolve_deps
resolver = Resolver.create(
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 460, in create
constraints, skipped, index_lookup, markers_lookup = resolver.get_metadata(
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 219, in get_metadata
req, req_idx, markers_idx = self.parse_line(
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 287, in parse_line
raise ResolutionFailure(
pipenv.exceptions.ResolutionFailure: ERROR: Failed to resolve requirement from line: https://files.pythonhosted.org/packages/8a/2a/fe2870e216511ed12b9b5acf89228abc0c76832b9572501e0b895a9e129d/spacy-3.4.3-cp39-cp39-macosx_10_9_x86_64.whl#egg=spacy-darwin#egg=[transformers]
✘ Locking Failed!

Steps to replicate

add below to pipfile:
'spacy = {file = "https://files.pythonhosted.org/packages/8a/2a/fe2870e216511ed12b9b5acf89228abc0c76832b9572501e0b895a9e129d/spacy-3.4.3-cp39-cp39-macosx_10_9_x86_64.whl", extras = ["transformers"]}'


Pipenv version: '2022.9.24'

Pipenv location: '/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv'

Python location: '/usr/local/Cellar/pipenv/2022.9.24/libexec/bin/python3.11'

OS Name: 'posix'

User pip version: '22.2.2'

PEP 508 Information:

{'implementation_name': 'cpython',
'implementation_version': '3.11.0',
'os_name': 'posix',
'platform_machine': 'x86_64',
'platform_python_implementation': 'CPython',
'platform_release': '22.1.0',
'platform_system': 'Darwin',
'platform_version': 'Darwin Kernel Version 22.1.0: Sun Oct 9 20:14:30 PDT '
'2022; root:xnu-8792.41.9~2/RELEASE_ARM64_T8103',
'python_full_version': '3.11.0',
'python_version': '3.11',
'sys_platform': 'darwin'}

@matteius
Copy link
Member

Can this be rechecked with pipenv==2022.12.17?

@matteius matteius added triage Type: Possible Bug This issue describes a possible bug in pipenv. labels Dec 17, 2022
@oz123
Copy link
Contributor

oz123 commented Dec 18, 2022

This still happens on the latest release.

 $ pipenv install -v
Pipfile.lock not found, creating...
Locking [packages] dependencies...
⠙ pipenv.exceptions.ResolutionFailure: ERROR: Failed to resolve requirement from line: 
$ pipenv --version
pipenv, version 2022.12.18.dev0
$ cat Pipfile 
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[dev-packages]

[packages]
spacy = {file = "https://files.pythonhosted.org/packages/8a/2a/fe2870e216511ed12b9b5acf89228abc0c76832b9572501e0b895a9e129d/spacy-3.4.3-cp39-cp39-macosx_10_9_x86_64.whl", extras = ["transformers"]}

[pipenv]
allow_prereleases = true

Looks like a bug in requirementslib.

@oz123 oz123 self-assigned this Dec 18, 2022
@oz123
Copy link
Contributor

oz123 commented Dec 18, 2022

Adding some more information I found out trying to debug this:

 $ .../virtualenvs/foo-_rgbUsyh/bin/python /...pypa/pipenv/pipenv/resolver.py --pre --category packages --write /tmp/resolverskea3pd8.json --constraints-file /tmp/pipenvy9wg0apmconstraints.txt
> ...pipenv/pipenv/resolver.py(802)_main()
-> os.environ["PIP_PYTHON_PATH"] = str(sys.executable)
(Pdb) c
>.../pypa/pipenv/pipenv/utils/resolver.py(273)parse_line()
-> try:
(Pdb) c
[ResolutionFailure]:   File "...pypa/pipenv/pipenv/resolver.py", line 738, in resolve
[ResolutionFailure]:       return resolve_deps(
[ResolutionFailure]:   File "...pypa/pipenv/pipenv/utils/resolver.py", line 1102, in resolve_deps
[ResolutionFailure]:       results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
[ResolutionFailure]:   File "...pypa/pipenv/pipenv/utils/resolver.py", line 890, in actually_resolve_deps
[ResolutionFailure]:       resolver = Resolver.create(
[ResolutionFailure]:   File "...pypa/pipenv/pipenv/utils/resolver.py", line 460, in create
[ResolutionFailure]:       constraints, skipped, index_lookup, markers_lookup = resolver.get_metadata(
[ResolutionFailure]:   File "...pypa/pipenv/pipenv/utils/resolver.py", line 216, in get_metadata
[ResolutionFailure]:       req, req_idx, markers_idx = self.parse_line(
[ResolutionFailure]:   File "...pypa/pipenv/pipenv/utils/resolver.py", line 287, in parse_line
[ResolutionFailure]:       raise ResolutionFailure(
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: Failed to resolve requirement from line: https://files.pythonhosted.org/packages/8a/2a/fe2870e216511ed12b9b5acf89228abc0c76832b9572501e0b895a9e129d/spacy-3.4.3-cp39-cp39-macosx_10_9_x86_64.whl#egg=spacy#egg=spacy[transformers]

@oz123
Copy link
Contributor

oz123 commented Dec 19, 2022

OK. This is definitely a bug in requirementslib:

(Pdb) p line
"{'https://files.pythonhosted.org/packages/8a/2a/fe2870e216511ed12b9b5acf89228abc0c76832b9572501e0b895a9e129d/spacy-3.4.3-cp39-cp39-macosx_10_9_x86_64.whl#egg=spacy#egg=spacy[transformers]"
(Pdb) s
--Call--
> /home/oznt/Software/pypa/pipenv/pipenv/vendor/requirementslib/models/requirements.py(2669)from_line()
-> @classmethod
(Pdb) n
> /home/oznt/Software/pypa/pipenv/pipenv/vendor/requirementslib/models/requirements.py(2673)from_line()
-> if isinstance(line, InstallRequirement):
(Pdb) n
> /home/oznt/Software/pypa/pipenv/pipenv/vendor/requirementslib/models/requirements.py(2675)from_line()
-> parsed_line = Line(line)
(Pdb) l
2670 	    @lru_cache()
2671 	    def from_line(cls, line):
2672 	        # type: (AnyStr) -> Requirement
2673 	        if isinstance(line, InstallRequirement):
2674 	            line = format_requirement(line)
2675 ->	        parsed_line = Line(line)

@oz123
Copy link
Contributor

oz123 commented Dec 19, 2022

Suspected cause of problem:

The constraints from:

spacy = {file = "https://files.pythonhosted.org/packages/8a/2a/fe2870e216511ed12b9b5acf89228abc0c76832b9572501e0b895a9e129d/spacy-3.4.3-cp39-cp39-macosx_10_9_x86_64.whl", extras = ["transformers"]}

are written to:

{'https://files.pythonhosted.org/packages/8a/2a/fe2870e216511ed12b9b5acf89228abc0c76832b9572501e0b895a9e129d/spacy-3.4.3-cp39-cp39-macosx_10_9_x86_64.whl#egg=spacy#egg=spacy[transformers] -i https://pypi.org/simple'}

@Blubberblub
Copy link

Blubberblub commented Dec 20, 2022

I stumbled over the same issue when trying to install a wheel for the ray library. From my pipfile:

ray = { extras = ['default'], file = "https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-3.0.0.dev0-cp310-cp310-manylinux2014_x86_64.whl" }

Gave me this error:

ERROR: Failed to resolve requirement from line: https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-3.0.0.dev0-cp310-cp310-manylinux2014_x86_64.whl#egg=ray#egg=ray[default]

Originally i tried to replicate this line for the installation through pip:

pip install "ray[default] @ https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-3.0.0.dev0-cp310-cp310-manylinux2014_x86_64.whl"

When installing the url through the command line without extras the installations works:

pipenv install https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-3.0.0.dev0-cp310-cp310-manylinux2014_x86_64.whl

pipenv, version 2022.12.19

oz123 added a commit that referenced this issue Dec 21, 2022
Requirements with a wheel path with extras caused a constraint line with
egg name which was included twice.
For example:

```
'spacy = {file = "https://files.pythonhosted.org/packages/.../spacy-3.4.3-cp39-..._x86_64.whl", extras = ["transformers"]}'
```

Produced a constraint line like the following:
```
https://files.pythonhosted.org/.../spacy-3.4.3-..._x86_64.whl#egg=spacy#egg=spacy[transformers]
```

This line triggered an unhandled excpetion `pipenv.exceptions.ResolutionFailure`,
as in #5536.

The should be submitted actually to requirementslib. However, I am
publishing it here so others can confirm it too.

Signed-off-by: Oz Tiram <[email protected]>
@oz123 oz123 added PR: awaiting-review The PR related to this issue is awaiting review by a maintainer. Status: In Progress This item is in progress. Category: Requirement and removed Type: Possible Bug This issue describes a possible bug in pipenv. triage labels Dec 21, 2022
@oz123
Copy link
Contributor

oz123 commented Dec 21, 2022

@elanou @Blubberblub I have published a branch with a commit that fixes this issue. Please see the detailed commit message, as for why this is still not merged. You can, however, test already the changes.
I will add an integration test for this here, and some unit tests in requirementslib. Once the upstream PR is merged, and vendored here, we can close this issue.
This is going to take a few more days. In the meanwhile, you can apply the patch to your version of pipenv and enjoy.

oz123 added a commit to sarugaku/requirementslib that referenced this issue Dec 21, 2022
Requirements with a wheel path with extras caused a constraint line with
egg name which was included twice.
For example:

```
'spacy = {file = "https://files.pythonhosted.org/packages/.../spacy-3.4.3-cp39-..._x86_64.whl", extras = ["transformers"]}'
```

Produced a constraint line like the following:
```
https://files.pythonhosted.org/.../spacy-3.4.3-..._x86_64.whl#egg=spacy#egg=spacy[transformers]
```

This line triggered an unhandled excpetion `pipenv.exceptions.ResolutionFailure`,
as in pypa/pipenv#5536.

Signed-off-by: Oz Tiram <[email protected]>
oz123 added a commit that referenced this issue Dec 28, 2022
Fix #5536.

Bump vendored requirementslib to 2.2.3 which includes the actual fix.
Also, add a test case for this in pipenv.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Category: Requirement PR: awaiting-review The PR related to this issue is awaiting review by a maintainer. Status: In Progress This item is in progress.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants