--require-hashes does not correctly handle pinned package with extras
#9644
Comments
|
One workaround for this is to use |
|
I think this might be the same issue I have encountered with pinned and hash-checking Broken requirements file: requirements.txt Steps to reproduce:Actual result:WorkaroundEither install Edit: I thought I had posted this as off-topic but after looking at the requirements.txt generated by pip-tools I see that it is specifying extras: |
|
If I understand correctly, #9775 did not fix this, but made it possible to fix this? |
|
Thanks @uranusjr. This is currently blocking CI and deployment for pypa/warehouse, is there any workaround? If not, could you explain at a high level what I'd need to do to fix this issue in pip? |
|
I believe it’s possible to work around this by manually “unpacking” extras, e.g. add a line to also provide hashes to |
|
I tried to create a failing test for this: def test_new_resolver_hash_with_extras(script):
parent_path = create_basic_wheel_for_package(
script, "parent", "0.1.0", depends=["child"]
)
parent_hash = hashlib.sha256(parent_path.read_bytes()).hexdigest()
child_path = create_basic_wheel_for_package(
script, "child", "0.1.0", extras={"extra": ["extra"]}
)
child_hash = hashlib.sha256(child_path.read_bytes()).hexdigest()
extra_path = create_basic_wheel_for_package(script, "extra", "0.1.0")
extra_hash = hashlib.sha256(extra_path.read_bytes()).hexdigest()
requirements_txt = script.scratch_path / "requirements.txt"
requirements_txt.write_text(
"""
child[extra]==0.1.0 --hash=sha256:{child_hash}
parent==0.1.0 --hash=sha256:{parent_hash}
extra==0.1.0 --hash=sha256:{extra_hash}
""".format(
child_hash=child_hash,
parent_hash=parent_hash,
extra_hash=extra_hash,
),
)
script.pip(
"install",
"--no-cache-dir",
"--no-index",
"--find-links", script.scratch_path,
"--requirement", requirements_txt,
)
assert_installed(script, parent="0.1.0", child="0.1.0", extra="0.1.0")but it's passing on the current main branch at e6414d6. Is it possible that this has already been fixed, or is my test not properly exercising this bug? |
|
I think the failure case should be like this: and change |
|
That still passes, but after looking more closely I was able to figure out what will cause it to fail:
It seems like what's happening is that I made a draft PR with the failing test here: #9995, but I don't have time at the moment to continue working on it to fix the issue. |
|
@di I have confirmed your theory by specifying another dependency without the extras package and everything installs pyjwt[crypto]==1.5.3 \
--hash=sha256:500be75b17a63f70072416843dc80c8821109030be824f4d14758f114978bae7 \
--hash=sha256:a4e5f1441e3ca7b382fd0c0b416777ced1f97c64ef0c33bfa39daf38505cfd2f
+pyjwt==1.5.3 \
+ --hash=sha256:500be75b17a63f70072416843dc80c8821109030be824f4d14758f114978bae7 \
+ --hash=sha256:a4e5f1441e3ca7b382fd0c0b416777ced1f97c64ef0c33bfa39daf38505cfd2f |
|
@cas-- It may be preferable to update all the dependencies that the extra includes to the latest versions instead. By removing the extra you are possibly removing sub-dependencies your application (or a dependency of your application) needs. |
Should be removed before merging pypi#8615
This reverts commit c6f664a.
|
@di Yeah all dependencies are specified with pip-tools as demonstrated in my previous comment. In my example it seems that child dependency |
|
@cas-- Ah, sorry, I missed that you were including both. That works too! |
- upstream issue: pypa/pip#9644 - runs tests within a container, uses the same `dockerfile` - ignores `pdm.lock`, removes `requirement.txt` - update manual contribution steps - bumps project version **Related Items** _Issues_ - Closes athul#137
- upstream issue: pypa/pip#9644 - runs tests within a container, uses the same `dockerfile` - ignores `pdm.lock`, removes `requirement.txt` - update manual contribution steps - bumps project version **Related Items** _Issues_ - Closes athul#137
- upstream issue: pypa/pip#9644 - runs tests within a container, uses the same `dockerfile` - ignores `pdm.lock`, removes `requirement.txt` - update manual contribution steps - bumps project version **Related Items** _Issues_ - Closes athul#137
|
FWIW #9995 did not fix the following case we encountered: one constraint with 3 extra, and another dependency constraint with 1 extra, subset of the previous 3 extra: With pip 23.2.1:
|
The tomli package is required by coverage (a dependency of pytest-cov
and coveralls), but the version is unpinned when upgrading to python
3.11.
`ERROR: In --require-hashes mode, all requirements must have their
versions pinned with ==. These do not:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 572.7/572.7 kB 86.5 MB/s eta 0:00:00
tomli from
https://files.pythonhosted.org/packages/97/75/10a9ebee3fd790d20926a90a2547f0bf78f371b2f13aa822c759680ca7b9/tomli-2.0.1-py3-none-any.whl
(from coverage[toml]==6.5.0->-r test-requirements.txt (line 173))`
This seems to be caused by a bug in pip [0], which results in
incorrect handling of extra dependencies. To work around this issue,
the tomli package version must be pinned.
[0] pypa/pip#9644
The tomli package is required by coverage (a dependency of pytest-cov
and coveralls), but the version is unpinned when upgrading to python
3.11.
`ERROR: In --require-hashes mode, all requirements must have their
versions pinned with ==. These do not:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 572.7/572.7 kB 86.5 MB/s eta 0:00:00
tomli from
https://files.pythonhosted.org/packages/97/75/10a9ebee3fd790d20926a90a2547f0bf78f371b2f13aa822c759680ca7b9/tomli-2.0.1-py3-none-any.whl
(from coverage[toml]==6.5.0->-r test-requirements.txt (line 173))`
This seems to be caused by a bug in pip [0], which results in
incorrect handling of extra dependencies. To work around this issue,
the tomli package version must be pinned.
[0] pypa/pip#9644
|
I am still getting this issue with urllib3 suddenly:
|
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as resolved.
This comment was marked as resolved.
This PR fixes an error while creating a local venv due to a bug in pip. It is also a follow-up to the second commit in #393. For more info, see pypa/pip#9644.
|
We started running into this issue yesterday and as far as I know we haven't updated the requirements.txt in over a month. The requirements.txt was created with the following command And then running the following fails with the below error. Interestingly I'm not able to reproduce this on my M1 Mac laptop, but is consistently occuring in our CI which uses a Linux python3.10 docker image. Also can reproduce locally inside the docker image. |
What did you want to do?
Install this requirements.txt file with pip 21.0.1 or master in a new virtual environment.
A package is pinned to a non-latest version with an extra (here,
requirements[security]) and another dependency requires this package without specifying the extra.Output
Install fails with:
The resolver does not recognise that
requests[security]==2.24.0fulfilsrequests<3, and tries to collect latestrequests. This fails as it is has no pinned hash.Additional information
Installation succeeds with
--use-deprecated=legacy-resolver.If the requirements file has no hashes, installation succeeds with the new resolver:
Collecting requests<3resolves to latest requests (2.25.1) in the install output, but the pinned version (2.24.0) is what ends up installed.If all packages are already installed in the environment,
pip installsucceeds (withRequirement already satisfied) even with the new resolver.The text was updated successfully, but these errors were encountered: