Poetry >=1.5.0 removes category from poetry.lock #7389

esev · 2023-06-03T03:11:00Z

Is there an existing issue for this?

I have searched the existing issues

Package ecosystem

pip (poetry)

Package manager version

poetry >=1.5.0

Language version

python 3.11

Manifest location and content before the Dependabot update

https://github.com/pywemo/pywemo/blob/main/poetry.lock

Previous format:

[[package]]
name = "cryptography"
version = "40.0.2"
description = "cryptography is a package which provides cryptographic recipes and primitives to Python developers."
category = "dev"
optional = false
python-versions = ">=3.6"
files = [
...

Format after Poetry 1.5.0 (lacks a category): pywemo/pywemo@65969c9 (poetry.lock)

[[package]]
name = "cryptography"
version = "40.0.2"
description = "cryptography is a package which provides cryptographic recipes and primitives to Python developers."
optional = false
python-versions = ">=3.6"
files = [
...

dependabot.yml content

version: 2
updates:
  - package-ecosystem: "pip"
    directory: "/"
    schedule:
      # Check for updates to Poetry dependencies every week
      interval: "weekly"

  - package-ecosystem: "github-actions"
    directory: "/"
    schedule:
      # Check for updates to GitHub Actions every week
      interval: "weekly"

Updated dependency

cryptography

What you expected to see, versus what you actually saw

Expected to see dependabot update for cryptography.

What happened:

Logs: https://github.com/pywemo/pywemo/security/dependabot/6/update-logs/336414827

dependabot-core/python/lib/dependabot/python/update_checker/poetry_version_resolver.rb

Lines 300 to 304 in d097b11

    
           def subdep_type 
        
             category = 
        
               TomlRB.parse(lockfile.content).fetch("package", []). 
        
               find { |dets| normalise(dets.fetch("name")) == dependency.name }. 
        
               fetch("category")

updater | 2023/06/02 20:34:03 INFO Raven 3.1.2 ready to catch errors
updater | 2023/06/02 20:34:04 INFO <job_672681628> Starting job processing
updater | 2023/06/02 20:34:08 INFO <job_672681628> Starting update job for pywemo/pywemo
updater | 2023/06/02 20:34:08 INFO <job_672681628> Checking if cryptography 40.0.2 needs updating
  proxy | 2023/06/02 20:34:08 [026] GET https://pypi.org:443/simple/cryptography/
  proxy | 2023/06/02 20:34:08 [026] 200 https://pypi.org:443/simple/cryptography/
updater | 2023/06/02 20:34:09 INFO <job_672681628> Latest version is 41.0.1
updater | 2023/06/02 20:34:13 INFO <job_672681628> Sending event 3c4a16f3d63f40c7baf2887883a8854e to Sentry
  proxy | 2023/06/02 20:34:13 [028] POST https://sentry.io:443/api/1451818/store/
  proxy | 2023/06/02 20:34:13 [028] 200 https://sentry.io:443/api/1451818/store/
updater | 2023/06/02 20:34:13 ERROR <job_672681628> Error processing cryptography (KeyError)
updater | 2023/06/02 20:34:13 ERROR <job_672681628> key not found: "category"
updater | 2023/06/02 20:34:13 ERROR <job_672681628> /home/dependabot/python/lib/dependabot/python/update_checker/poetry_version_resolver.rb:304:in `fetch'
updater | 2023/06/02 20:34:13 ERROR <job_672681628> /home/dependabot/python/lib/dependabot/python/update_checker/poetry_version_resolver.rb:304:in `subdep_type'
updater | 2023/06/02 20:34:13 ERROR <job_672681628> /home/dependabot/python/lib/dependabot/python/update_checker/poetry_version_resolver.rb:281:in `set_target_dependency_req'
updater | 2023/06/02 20:34:13 ERROR <job_672681628> /home/dependabot/python/lib/dependabot/python/update_checker/poetry_version_resolver.rb:232:in `updated_pyproject_content'
updater | 2023/06/02 20:34:13 ERROR <job_672681628> /home/dependabot/python/lib/dependabot/python/update_checker/poetry_version_resolver.rb:214:in `write_temporary_dependency_files'
...

I've also noticed dependabot adding the category to poetry.lock on PRs. category should not have been added after updating Poetry to 1.5.0.

Native package manager behavior

The category field is not present in poetry.lock

See python-poetry/poetry#7637 where category was removed.

Images of the diff or a link to the PR, issue, or logs

Logs: https://github.com/pywemo/pywemo/security/dependabot/6/update-logs/336414827

Smallest manifest that reproduces the issue

No response

The text was updated successfully, but these errors were encountered:

phillipuniverse · 2023-06-09T14:27:23Z

@landongrindheim I'm not so sure this is complete by simply updating Poetry, see #7418.

EDIT - yeah same problem as now caught by #7418, this issue should be re-opened.

n-thumann · 2023-06-12T07:46:03Z

This might still be an issue, because in greenbone/troubadix#578 Dependabot removed the category of all dependencies.
Now, when re-running Dependabot, it failed with Dependabot encountered an unknown error and the following (excerpt) showing up in the log:

[...]
updater | 2023/06/12 07:11:59 INFO <job_676691621> No update possible for pontos 23.3.5
updater | 2023/06/12 07:11:59 INFO <job_676691621> Checking if markdown-it-py 2.2.0 needs updating
  proxy | 2023/06/12 07:11:59 [232] GET https://pypi.org:443/simple/markdown-it-py/
  proxy | 2023/06/12 07:11:59 [232] 200 https://pypi.org:443/simple/markdown-it-py/
updater | 2023/06/12 07:11:59 INFO <job_676691621> Latest version is 3.0.0
updater | 2023/06/12 07:12:00 INFO <job_676691621> Sending event 1e66ef70eefd47f3866a80e81ad7cf9c to Sentry
  proxy | 2023/06/12 07:12:00 [234] POST https://sentry.io:443/api/1451818/store/
  proxy | 2023/06/12 07:12:00 [234] 200 https://sentry.io:443/api/1451818/store/
updater | 2023/06/12 07:12:00 ERROR <job_676691621> Error processing markdown-it-py (KeyError)
updater | 2023/06/12 07:12:00 ERROR <job_676691621> key not found: "category"
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/python/lib/dependabot/python/update_checker/poetry_version_resolver.rb:304:in `fetch'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/python/lib/dependabot/python/update_checker/poetry_version_resolver.rb:304:in `subdep_type'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/python/lib/dependabot/python/update_checker/poetry_version_resolver.rb:281:in `set_target_dependency_req'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/python/lib/dependabot/python/update_checker/poetry_version_resolver.rb:232:in `updated_pyproject_content'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/python/lib/dependabot/python/update_checker/poetry_version_resolver.rb:214:in `write_temporary_dependency_files'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/python/lib/dependabot/python/update_checker/poetry_version_resolver.rb:92:in `block (2 levels) in fetch_latest_resolvable_version_string'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/common/lib/dependabot/shared_helpers.rb:181:in `with_git_configured'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/python/lib/dependabot/python/update_checker/poetry_version_resolver.rb:91:in `block in fetch_latest_resolvable_version_string'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/common/lib/dependabot/shared_helpers.rb:49:in `block in in_a_temporary_directory'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/common/lib/dependabot/shared_helpers.rb:49:in `chdir'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/common/lib/dependabot/shared_helpers.rb:49:in `in_a_temporary_directory'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/python/lib/dependabot/python/update_checker/poetry_version_resolver.rb:90:in `fetch_latest_resolvable_version_string'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/python/lib/dependabot/python/update_checker/poetry_version_resolver.rb:62:in `latest_resolvable_version'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/python/lib/dependabot/python/update_checker.rb:42:in `latest_resolvable_version'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/common/lib/dependabot/update_checkers/base.rb:74:in `preferred_resolvable_version'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/common/lib/dependabot/update_checkers/base.rb:260:in `preferred_version_resolvable_with_unlock?'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/common/lib/dependabot/update_checkers/base.rb:252:in `numeric_version_can_update?'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/common/lib/dependabot/update_checkers/base.rb:202:in `version_can_update?'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/common/lib/dependabot/update_checkers/base.rb:44:in `can_update?'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/dependabot-updater/lib/dependabot/updater/operations/update_all_versions.rb:204:in `requirements_to_unlock'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/dependabot-updater/lib/dependabot/updater/operations/update_all_versions.rb:88:in `check_and_create_pull_request'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/dependabot-updater/lib/dependabot/updater/operations/update_all_versions.rb:59:in `check_and_create_pr_with_error_handling'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/dependabot-updater/lib/dependabot/updater/operations/update_all_versions.rb:34:in `block in perform'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/dependabot-updater/lib/dependabot/updater/operations/update_all_versions.rb:34:in `each'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/dependabot-updater/lib/dependabot/updater/operations/update_all_versions.rb:34:in `perform'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:72:in `run'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/dependabot-updater/lib/dependabot/update_files_command.rb:38:in `perform_job'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/dependabot-updater/lib/dependabot/base_command.rb:52:in `run'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> bin/update_files.rb:23:in `<main>'
updater | 2023/06/12 07:12:00 INFO <job_676691621> Checking if colorama 0.4.6 needs updating
[...]

We were also able to reproduce this behavior in another private repository and notice it, because @dependabot rebase failed.

phillipuniverse · 2023-06-21T14:38:56Z

More information on potential resolution added at #7418 (comment).

@esev @n-thumann - trying to figure out why this breaks in some flows but not in others. In all of my projects Dependabot distributes updates correctly. #7418 replicates the scenario in a test, but how does this work in practice? Is it that Dependabot breaks when only running with a poetry.lock file and the absence of pyproject.toml?

esev · 2023-06-21T15:24:06Z

I have both a poetry.lock and a pyproject.toml file. The only time I saw this error was for a Dependabot update via the Github Dependabot Vulnerability alerts under the Security tab. I continue to receive the normal PRs from Dependabot without any issues.

phillipuniverse · 2023-06-21T16:02:41Z

Ah! Perhaps Dependabot security checks only operate in the context of a poetry.lock. Whenever I see one of these it does always mention that the problem was in poetry.lock, not pyproject.toml.

Or maybe more likely - in the case of cryptography that usually comes in transitively, so maybe it was never managed in a pyproject.toml to begin with, and Dependabot only considers pyproject.toml if it needs to

esev · 2023-06-21T22:58:56Z

Or maybe more likely - in the case of cryptography that usually comes in transitively, so maybe it was never managed in a pyproject.toml to begin with, and Dependabot only considers pyproject.toml if it needs to

Interesting! I hadn't made that connection, but now that you mention it, I haven't gotten any PRs to update transitive dependencies recently

matthias-bach-by · 2023-06-22T08:15:39Z

That is a great observation. All of the dependencies for which Dependabot is failing me are transitive, while I keep getting updates for all my direct dependencies.

There is one important outlier, though. The PR that did remove the categories from my poetry.lock file was also a transitive one. But that was the last transitive update to succeed.

n-thumann · 2023-06-22T08:41:16Z

All of the dependencies for which Dependabot is failing me are transitive, while I keep getting updates for all my direct dependencies.

We noticed the same behavior: Even though the category was removed in greenbone/troubadix#578 and a we see the error from my comment above ever since, Dependabot was able to update a (direct, non-transitive) dependency in greenbone/troubadix#584 and updated poetry.lock accordingly.

Additionally, I checked the latest output and it contains:

updater | Dependabot encountered '12' error(s) during execution, please check the logs for more details.
updater | +------------------------------------+
updater | |   Dependencies failed to update    |
updater | +--------------------+---------------+
updater | | isort              | unknown_error |
updater | | astroid            | unknown_error |
updater | | semver             | unknown_error |
updater | | pylint             | unknown_error |
updater | | platformdirs       | unknown_error |
updater | | markdown-it-py     | unknown_error |
updater | | rich               | unknown_error |
updater | | setuptools         | unknown_error |
updater | | httpx              | unknown_error |
updater | | httpcore           | unknown_error |
updater | | importlib-metadata | unknown_error |
updater | | rfc3986            | unknown_error |
updater | +--------------------+---------------+

All of these 12 dependencies are transitive (checked with poetry show --tree).

Dependabot has a compatibility issue with Poetry >= 1.5.0 due to removal of the `category` key from `poetry.lock`: dependabot/dependabot-core#7389 Regenerate the lock file with Poetry 1.4.2 as a temporary workaround.

Dependabot has a compatibility issue with Poetry >= 1.5.0 due to removal of the `category` key from `poetry.lock`: dependabot/dependabot-core#7389 Regenerate the lock file with Poetry 1.4.2 as a temporary workaround. This needs to be repeated for every PR generated by Dependabot.

sigprof · 2023-07-04T06:55:01Z

Running poetry lock --no-update with Poetry 1.4.2 currently works around the problem — Dependabot is able to detect the needed updates and open the corresponding PRs. But each of those PRs updates the poetry.lock file to the new format again, so the same change needs to be added to every such PR.

phillipuniverse · 2023-07-06T04:51:46Z

There is one important outlier, though. The PR that did remove the categories from my poetry.lock file was also a transitive one. But that was the last transitive update to succeed.

@matthias-bach-by this outlier makes sense. Dependabot checked for updates when poetry.lock was still on the pre-1.5 version, which included the category key. When Dependabot updated poetry.lock in its normal update process, Dependabot was running with Poetry 1.5 which removed the category key.

This is becoming more of a problem I got a recent CVE on grpcio and Dependabot could not generate a security update because of this transitive dependency issue (grpcio comes in for me transitively from otel).

I am fairly confident that the solution will be in #7418 but I need more input from the Dependabot maintainers to proceed.

Dependabot has a compatibility issue with Poetry >= 1.5.0 due to removal of the `category` key from `poetry.lock`: dependabot/dependabot-core#7389 Regenerate the lock file with Poetry 1.4.2 as a temporary workaround. This needs to be repeated for every PR generated by Dependabot.

rachelwigell · 2023-08-25T13:48:07Z

Hello hello! We have been having this issue too with the category lines being removed in our dependabot PRs in our repo where we use poetry. I see that v0.228.0 of dependabot-core was released yesterday with the fix from #7834 in it. I just rebased an open dependabot PR we had, and I'm able to confirm that the new poetry.lock file was generated on v0.228.0 because it says poetry 1.6.1 was used. However, I'm still seeing that the category lines are removed. Is anyone else still having issues with this today?

deivid-rodriguez · 2023-08-25T14:00:29Z

Hei! Unfortunately this is an existing issue, yes.

We currently don't respect the poetry version your poetry.lock uses, but instead use our own fixed version of poetry (1.6.1). This version of Poetry removes category lines from the lockfile when it finds them.

You can track resolution of this issue at #1556.

phillipuniverse · 2023-08-25T15:19:05Z

However, I'm still seeing that the category lines are removed. Is anyone else still having issues with this today?

@rachelwigell and just to be clear - the category key being removed is not a bug in and of itself, this is expected behavior when Poetry 1.5+ generates a lock file.

rachelwigell · 2023-08-25T15:20:14Z

Thanks both! I had misunderstood the issue. We can likely upgrade poetry in our repo to stop these huge diffs from being created.

esev added the T: bug 🐞 Something isn't working label Jun 3, 2023

jakecoffman added the L: python:poetry Python packages via poetry label Jun 6, 2023

landongrindheim mentioned this issue Jun 8, 2023

Update poetry requirement from <1.4.0,>=1.1.15 to >=1.1.15,<1.6.0 in /python/helpers #7350

Merged

landongrindheim closed this as completed in #7350 Jun 8, 2023

phillipuniverse mentioned this issue Jun 9, 2023

Support Poetry 1.5.1 poetry.lock with absence of category key #7418

Closed

landongrindheim reopened this Jun 12, 2023

jonathangreen mentioned this issue Jul 17, 2023

Update cryptography ThePalaceProject/circulation#1277

Merged

2 tasks

This was referenced Jul 20, 2023

chore(deps): bump cryptography from 39.0.1 to 41.0.2 artsy/hokusai#358

Closed

chore(deps-dev): bump pygments from 2.14.0 to 2.15.0 artsy/hokusai#359

Closed

justinpolygon mentioned this issue Jul 23, 2023

Bump sphinx-rtd-theme from 1.1.1 to 1.2.2 polygon-io/client-python#458

Merged

bayandin mentioned this issue Jul 26, 2023

Bump certifi from 2022.12.7 to 2023.7.22 neondatabase/neon#4815

Merged

5 tasks

n-thumann mentioned this issue Jul 27, 2023

Require Python 3.9+ and update dependencies greenbone/troubadix#594

Merged

1 task

bayandin mentioned this issue Aug 2, 2023

Bump cryptography from 41.0.2 to 41.0.3 neondatabase/neon#4870

Merged

5 tasks

jonathangreen mentioned this issue Aug 3, 2023

Bump cryptography to 41.0.3 ThePalaceProject/circulation#1305

Merged

2 tasks

jamesturner246 mentioned this issue Aug 3, 2023

Remove deprecated pytest-flake8. ImperialCollegeLondon/pycsvy#66

Merged

deivid-rodriguez mentioned this issue Aug 17, 2023

Add support for Poetry 1.5 lockfiles #7834

Merged

jeffwidman closed this as completed in #7834 Aug 19, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Poetry >=1.5.0 removes category from poetry.lock #7389

Poetry >=1.5.0 removes category from poetry.lock #7389

esev commented Jun 3, 2023

phillipuniverse commented Jun 9, 2023 •

edited

Loading

n-thumann commented Jun 12, 2023

phillipuniverse commented Jun 21, 2023

esev commented Jun 21, 2023 •

edited

Loading

phillipuniverse commented Jun 21, 2023 •

edited

Loading

esev commented Jun 21, 2023 •

edited

Loading

matthias-bach-by commented Jun 22, 2023

n-thumann commented Jun 22, 2023

sigprof commented Jul 4, 2023

phillipuniverse commented Jul 6, 2023

rachelwigell commented Aug 25, 2023

deivid-rodriguez commented Aug 25, 2023 •

edited

Loading

phillipuniverse commented Aug 25, 2023

rachelwigell commented Aug 25, 2023

Poetry >=1.5.0 removes category from poetry.lock #7389

Poetry >=1.5.0 removes category from poetry.lock #7389

Comments

esev commented Jun 3, 2023

Is there an existing issue for this?

Package ecosystem

Package manager version

Language version

Manifest location and content before the Dependabot update

dependabot.yml content

Updated dependency

What you expected to see, versus what you actually saw

Native package manager behavior

Images of the diff or a link to the PR, issue, or logs

Smallest manifest that reproduces the issue

phillipuniverse commented Jun 9, 2023 • edited Loading

n-thumann commented Jun 12, 2023

phillipuniverse commented Jun 21, 2023

esev commented Jun 21, 2023 • edited Loading

phillipuniverse commented Jun 21, 2023 • edited Loading

esev commented Jun 21, 2023 • edited Loading

matthias-bach-by commented Jun 22, 2023

n-thumann commented Jun 22, 2023

sigprof commented Jul 4, 2023

phillipuniverse commented Jul 6, 2023

rachelwigell commented Aug 25, 2023

deivid-rodriguez commented Aug 25, 2023 • edited Loading

phillipuniverse commented Aug 25, 2023

rachelwigell commented Aug 25, 2023

phillipuniverse commented Jun 9, 2023 •

edited

Loading

esev commented Jun 21, 2023 •

edited

Loading

phillipuniverse commented Jun 21, 2023 •

edited

Loading

esev commented Jun 21, 2023 •

edited

Loading

deivid-rodriguez commented Aug 25, 2023 •

edited

Loading