Regression in namespace package metadata on poetry-core 1.1.0rc1

[ariebovenberg]

• I am on the latest Poetry version. n/a
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option). n/a

• OS version and name: MacOS 12.5.1
• Poetry version: poetry-core 1.1.0rc1
• Link of a Gist with the contents of your pyproject.toml file: see below

Issue

An addition in poetry-core between 1.1.0a7 and 1.1.0b1 introduced a bug in package metadata name for namespaced packages:

>>> from importlib import metadata
>>> metadata('mynamespace.foomodule')
importlib.metadata.PackageNotFoundError: mynamespace.foomodule

The package is instead registered as mynamespace-foomodule. This is a particular problem since importlib.metadata is the canonical way to single-source the __version__

This bug occurs on Python 3.8 and 3.9 (but not 3.10 insterestingly enough). The change that introduced the bug: python-poetry/poetry-core#328. Ever since this change, . is normalized to - in package names, which is incorrect for namespace packages.

To reproduce

pyproject.toml

[tool.poetry]
name = "mynamespace.foomodule"
version = "0.1.0"
description = ""
authors = ["Your Name <[email protected]>"]
readme = "README.md"
packages = [
    { include = "mynamespace"},
]

[tool.poetry.dependencies]
python = "^3.8"

[build-system]
requires = ["poetry-core==1.1.0rc1"]
build-backend = "poetry.core.masonry.api"

files

mynamespace/
├── __init__.py  # contains: __path__ = __import__('pkgutil').extend_path(__path__, __name__)
└── foomodule
    └── __init__.py  # empty

Then, run:

# make sure you're on python 3.8 or 3.9
pip install .
# this raises an error, but shouldn't
python -c "from importlib import metadata; metadata.version('mynamespace.foomodule')"

What to do?

I don't mind submitting a fix in poetry-core, but I'm of course missing a lot of context on why the change was made in the first place.

[dimbleby]

Would be interested in a link to the python 3.10 MR / discussion around the changed behaviour, if you can dig it out.

My insta-take is that poetry is not wrong here: would be interesting to see whether that upstream discussion is framed as "some callers do it wrong but we can help" or as "oops, we should work with canonicalized package names". I'm hoping for the latter.

Meanwhile, can you use canonicalized package names all along? The examples at https://packaging.python.org/en/latest/guides/packaging-namespace-packages/ use mynamespace-subpackage-a (not mynamespace.subpackage.a), suggesting that this was always the preferred approach.

[ariebovenberg]

@dimbleby thanks for the info, digging into the background I now understand the reasoning behind canonicalization, and I can understand heading in that direction.

Good find in the namespace packaging docs! Using name = "mynamespace-foomodule" in pyproject.toml bypasses the entire issue.

[PeterJCLaw]

@dimbleby given the frequency that this issue has appeared and that fact that it ends up being breaking (or very nearly so) for a number of use cases, would you consider adding a warning from poetry (ideally with a link to an explanation and how to fix) when it does this canonicalisation? Alternatively I would encourage that poetry actually just reject these names rather than silently change them in ways that aren't always compatible with previous versions of the same package.

It feels like either of those would ensure that the issue is visible when the offending package is built, rather than only once it has already been published and users are trying to use it.

A major version bump for this arguably breaking change might have been worth considering too (though I realise there's not much can be done about that now).

---

Aside: In my case we actually had issues uploading it to PyPI which seemed to stem from the source distribution keeping the dot, while the wheel didn't (though we weren't able to reproduce that failure mode); then confusion when PyPI had seemed to change the name of the project unexpectedly. In the end we pinned an older version of poetry and yanked the version of the package with the adjusted name in order to avoid breakage for now. I realise that's not a permanent solution of course. poetry also seemed to be using underscores rather than dashes in the artefact names, which seems to disagree with canonicalisation to dashes and further confused us.

[dimbleby]

Doubtful. If you want to offer an MR then I expect it would get proper consideration, but I've no interest in doing this myself.

If you want to be able to upload packages to pypi with dots in their names, please go encourage pypi/warehouse#10030. If and when that is resolved at pypi, it will be straightforward for poetry to follow.

[PeterJCLaw]

If you want to be able to upload packages to pypi with dots in their names, please go encourage pypi/warehouse#10030. If and when that is resolved at pypi, it will be straightforward for poetry to follow.

For clarity: part of what causes confusion for me on this is that PyPI demonstrably does accept distribution files with dots in (e.g: sr.robot3) when the package name also contains dots.

I (now) realise there are PEPs which suggest that package names shouldn't contain dots (though the discussion on the practicalities of that and where those PEPs apply seems far from resolved), however that doesn't change the reality that dots are being used in the wild and this change was breaking for a number of projects.

The fact that PyPI doesn't accept doted package names to have underscored file names seems like a separate issue to me.

I'll have a look at putting together some changes; does poetry have a warnings framework I should be looking to use? From a quick look I can see there's use of the standard library logging module which seems to line up with the printed output (plus some colourisation I've not found), is that the way to go here?

[neersighted]

The issue is that PyPI accepts dotted names, but not in a consistent way. We can't programmatically normalize a name to be valid for PyPI and allow dots/capitalization, and we don't want to leak PyPI's idiosyncratic validation into Poetry code. See #1202 for more of this.

Certainly, you're welcome to add a warning when names do get normalized and may surprise the user when they go to reference them -- that being said, I don't think Poetry should force people to use normalized names in their pyproject.toml -- it's just that the artifacts we produce need to be normalized as this causes less problems than not normalizing as we did before (there were a lot of ugly bugs around this).

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.