Package names containing a dot don't work under DevPI #3666
Comments
|
I tried setting up a local installation of devpi-server 3.1.0 and the same problem still occurs. |
|
I met this issue as well, I checked the source code, every time pip finding packages, the package name is passed to a function called That function is located at Regarding to zope.interface, seems both versions (zope.interface and zope-interface) are available in https://pypi.python.org/simple/, that's why we can download zope.interface via pypi.python.org. I doubt if converting dot to dash is intentional. anyway , a change might need on either pip or devpi-server |
|
We've seen it too. I hope this gets fixed soon. |
|
Same here with python 3.4 |
|
I'm seeing the same issue with libraries hosted on artifactory, so this goes beyond devpi and pypi. |
|
The tl;dr is that these tools will need to be updated to match the normalization scheme as defined in PEP 503. This can be implemented using the packaging library's The longer explanation is that originally the simple API was not well defined but PyPI had all of the URLs using the form of the name that the author originally entered (e.g. Django not django) and the various tools (pip / easy_install) would make a request using exactly what the use typed (e.g. pip install django requested it as 'django' not Django). This generally worked fine on PyPI because PyPI would redirect anything that matched what became the PEP 503 normalization rules to the correct thing. Obviously a static mirror like bandersnatch couldn't do this redirect so if pip/easy_install couldn't find a name it would fall back to requesting the top level API that lists every single package on PyPI. This worked but it meant that people were regularly downloading a several mb file from Mirrors because they happened to use a slightly different but equivalent version of the name. It also caused extra http requests even in the PyPI case to redirect people to the correct place. To remedy this I wrote PEP 503 to define the simple API and instead of making the author provided name the cannonical name I made the normalized name the cannonical name which meant that we could do the normalization everywhere without having to worry about redirects (except for old versions of the tools) and without the nasty several mb fallback. This was implemented in pip with version 8. However due to an oversight the '.' Was not correctly getting normalized by pip (and a lot of other tools) but because a lot of them mis-implemented then similarly it worked. However with pip 8.1.2 we updated one of our dependencies which correctly implemented this and that caused pip to start correctly normalizing the '.' but until these tools like devpi get updated things aren't going to quite work. As a transitional thing tools should probably support both the erroneously normalized and the correctly normalized form. Sent from my iPhone
|
|
This is pretty bad. Every single installation of devpi and similar tools will now need to be upgraded immediately. Who knows, some projects might not be able to use the latest (fixed) version of such a tool for other reasons, and the fix likely won't be backported to every old version of those tools. Perhaps some kind of deprecation timeline would have been possible, to give the authors of those tools time to at least create a fixed version? |
|
For anyone using tox on Travis CI, which creates a virtualenv and installs the latest pip (even though Travis CI environment itself uses pip 6.0.7, here's how we worked around this problem. Add a #!/bin/bash
pip install '<8.1.2'
pip install "$@"Add to your This will downgrade the version of pip inside the virtualenv created by tox, before installing the dependencies. |
|
Well, PEP 503 was published 8 months ago, so tools like devpi have had some time to implement it. It's a shame they've been slow in doing so, but I'm not sure what we can do about that (other than raising an issue on devpi). |
|
Still running tests, but it looks like a devpi-common 2.0.9 release will be enough to fix this. |
|
@pfmoore well, imho that bugfix should have been a major release (from my pov its a breaking change of interaction with external systems) |
|
I bisected this problem to 8e236dd |
|
@RonnyPfannschmidt Major release for what? I believe from what @dstufft said the support was added in pip in the version 8 (a major release) - there was a bug around the dot character in 8.0 itself, that bugfix was in a minor release but that seems perfectly fine to me. Or do you mean a devpi major release? |
|
@pfmoore im aware that the code change is small, but the behavior change is breaking as such my opinion is, that a major release might have been better so people notice that the update might break something as for devpi, i don't know if the change on the server side is breaking or just cosmetic |
|
@RonnyPfannschmidt Well, the behaviour change was in 8. It just didn't work properly. Would you have been happier if pip 8 had rewritten Users who were really carefully following the feature progression would have seen that PEP 503 was in pip 8, and that devpi hadn't implemented the server side of it, and pinned to |
|
Whether or not this change should have triggered a major or minor or path release would be a more interesting question if anybody had noticed that pip 8 didn't correctly normalize the |
|
IOW, nobody made a conscious choice to change this behavior in 8.1.2, it was a consequence of the fact that I expected any changes like this to have already happened, so I didn't look real close and our dependencies updating to PEP 503 behavior. |
|
@pfmoore my understanding is that the intent was to change behavior in pip 8 and the fix life-cycle accidentally deferred that to 8.1.2 i'd prefer to only see that kind of change for major version jumps (as the intent was) i'm well aware the current problem is unintended and i'm pretty sure i'd have done the release just as @dstufft did this is more of a "ouch this is an accident, it should be major due to breakage we observe" |
|
@pfmoore semantic correctness doesn't always trump pragmatic real world concerns. Unfortunately, I am not at all surprised that a tonne of projects that interact with pypi had no idea that pip 8 introduced this change or that there was a bug in it. Widely used projects do often consider the actual real world impact on fixing "bugs" that result in backwards incompatible breakage for 3rd party code. I'm not sure if it would have been possible, but perhaps making this a soft change in 8.0 and monitoring the tools to find problems like this before enabling it could have prevented unexpected breakage. |
|
To be clear it was warning since pip 6.0 in the general case, it was an oversight that it wasn't correctly converting a Unfortunately we don't have any metrics into how people are invoking pip, what the outcomes of their commands are, or anything like that which we could use to judge impact (if we had even realized this wasn't working correctly in the first place or that we had inadvertently fixed it) so impact tends to be judged based on gut feeling mostly and testing against PyPI itself. Prior to PEP 503 the definition of what a repository was, was literally "whatever PyPI does" so all the non PyPI implementations attempted to copy that to varying degrees of success and typically they didn't re-evaluate that when PyPI changed until Typically I would roll this back and issue a 8.1.3 and send this specific change through some kind of deprecation process... however, the dependency we updated and the changes that caused it to actually happen are fixing other important bugs caused by the new marker support. Rolling that back will re-break some packages that are currently not able to be installed by pip 8 -> 8.1.1. It ends up being a choice between breaking those packages, or breaking things like devpi and bandersnatch, and I have to fall along the axis of fixing the things we meant to fix in 8.1.2 and letting the breaking change stay, largely because those packages don't have much of a recourse for fixing it otherwise and this breaking change can be fixed in devpi/bandersnatch/etc instead. |
|
See also, Postel's Tarpit. |
|
@dstufft thanks for the additional explanation. That all makes sense and sounds like the right thing has been done in the end, it was just unfortunate that tools like devpi had not been updated already. I'm not developing those tools, I'm only using them, so your explanation adds some clarity to the situation for me personally. |
|
devpi fix is out, see https://groups.google.com/d/msg/devpi-dev/-MgysUfZ36Q/JnqqMscrAAAJ |
|
@fschulze thanks for the quick fix :) |
|
@dstufft - sadly, no one except the few dedicated souls like your good self actually read, let alone understand PEPs (for this one you'd have had to be on the pypa lists (distutils-sig, still?) or maybe python-dev? to even find out about the PEP - is that a realistic expectation for everyone who runs any python package hosting infrastructure?). The rest of us react to stuff changing as we find out about it, most often when things break. What makes me sad is that this change means you can no longer host a poor man's python package repository using a filesystem and apache, unless you can guarantee no packages include dots in their names... |
|
@fschulze - any chance of a point release fix for devpi-server 2.x too? |
|
@cjw296 you can, by either depending on pip >= 8.1.2 or adding a rewrite rule to your webserver. Store the files with dashes and rewrite anything with [_.]+ to _. And if you had read the link, you would have seen that devpi-common 2.0.9 fixes the issue for server 2.6.x as well. |
|
@cjw296 I expect that the people who are writing things like devpi and such will be on That being said, you can still trivially host a "poor man's repository" in one of two ways:
|
|
And I'm sure only developers have been affected by this pip change anyway, because in production everyone pins all their packages including pip, right? (SCNR) |
|
Firstly, having NEVER experienced a warning in regard to "." in package names, this effective pulling the plug is pretty bad. Secondly, while PyPI still has packages using the "." pip should allow for it. Goto https://pypi.python.org/pypi/zope.interface vs https://pypi.python.org/pypi/zope-interface Notice where you redirect in the 2nd case. I must say I believe PEP 503's forcing of "-" to be the effective "separator" in package names to be a cause of confusion, if not simply wrong. Reasons:
|
|
We obviously can't replace the release notes in the already released artifact, but if you want to submit a PR to master adding the changelog item we can update them for the website and the next release. |
|
This seems to be breaking pypiserver ( https://pypi.python.org/pypi/pypiserver ) as well... even the latest version. Might wanna check back with the devs to get it working for everyone again |
ghost
mentioned this issue
|
Faced this problem with pip 8.1.2 and my company's devpi repo (devpi-server-2.1.5.post3 Looks like it's a problem with pip 8., 'cause with 7. everything is ok. |
|
pip 8.1.1 doesn't have this problem. |
|
@chadawagner thank you! |
According to PEP 503 a "normalized" project name has all runs of the characters ., - and _ replaced with a single - character.[0] Similarly, since version 8.1.2, that is the behaviour of pip as well. [1] However, Setuptools still allows a . in the normalized name, which is causing trouble down the line. [0] https://www.python.org/dev/peps/pep-0503/#normalized-names [1] pypa/pip#3666
According to PEP 503, a "normalized" project name has all runs of the characters ., - and _ replaced with a single - character.[0] Similarly, since version 8.1.2, that is the behaviour of pip as well. [1] However, Setuptools still allows a . in the normalized name, which is causing trouble down the line. [0] https://www.python.org/dev/peps/pep-0503/#normalized-names [1] pypa/pip#3666
According to PEP 503, a "normalized" project name has all runs of the characters ., - and _ replaced with a single - character.[0] Similarly, since version 8.1.2, that is the behaviour of pip as well. [1] However, Setuptools still allows a . in the normalized name, which is causing trouble down the line. [0] https://www.python.org/dev/peps/pep-0503/#normalized-names [1] pypa/pip#3666
According to PEP 503, a "normalized" project name has all runs of the characters ., - and _ replaced with a single - character. [0] Similarly, since version 8.1.2, that is the behaviour of pip as well. [1] However, Setuptools still allows a . in the normalized name, which is causing trouble down the line. [0] https://www.python.org/dev/peps/pep-0503/#normalized-names [1] pypa/pip#3666
According to PEP 503, a "normalized" project name has all runs of the characters ., - and _ replaced with a single - character. [0] Similarly, since version 8.1.2, that is the behaviour of pip as well. [1] However, Setuptools still allows a . in the normalized name, which is causing trouble down the line. [0] https://www.python.org/dev/peps/pep-0503/#normalized-names [1] pypa/pip#3666
lock
bot
added
the
auto-locked
Description:
When asked for a package name containing a dot version 8.1.2 of pip rewrites the package name with a dash instead (e.g. "foo.bar" becomes "foo-bar"). This seems to work fine on PyPI, but not on DevPI (we're using devpi-server-2.1.4). Downgrading to version 8.1.1 of pip resolves the problem.
What I've run:
The text was updated successfully, but these errors were encountered: