pip 21 rejects wheels with local version labels

[takluyver]

PR #9320 added some metadata validation of wheels built through pip. People using local version identifiers (e.g. 0.1+deadbeef) reported bugs in Flit because it builds wheels which pip rejects based on the filename, like this:

Building wheels for collected packages: testpkg
  Building wheel for testpkg (PEP 517) ... done
  Created wheel for testpkg: filename=testpkg-0.6.1.dev5_ga652c20-py3-none-any.whl size=1181 sha256=cb8856b9035b1cf8021533a8ce7a07f02402c942603ce53e17a5ceea7e9cd787
  Stored in directory: /home/sturm/.cache/pip/wheels/34/c8/42/38155569da3179727ba8f011d468b6a0e6f789986ec22ae3d0
  WARNING: Built wheel for testpkg is invalid: Wheel has unexpected file name: expected '0.6.1.dev5+ga652c20', got '0.6.1.dev5-ga652c20'
Failed to build testpkg

I believe Flit is complying with the spec, specifically PEP 427's section on escaping in the wheel filename:

Each component of the filename is escaped by replacing runs of non-alphanumeric characters with an underscore _:
re.sub("[^\w\d.]+", "_", distribution, re.UNICODE)

I understand that to mean that a 0.1+deadbeef version number should make a .whl filename containing -0.1_deadbeef-.Pip appears to reject that, and I believe it's pip that has this wrong. PyPUG still says that PEP 427 is the sole specification for the wheel format.

If the outcome is that the spec changes to match what pip does, that's fine, and I will fix Flit accordingly. But I don't want to assume that's the case and merge something that goes against the current spec.

[uranusjr]

There was actually a discussion on this a while ago. The problem with escaping + to _ is that the escaped value cannot be converted back to its original value, thus rendered useless. Quoting pypa/wheel#269 (comment):

That part of the spec is probably a mistake where it says "all components of the filename..."

Given Daniel is the author of the wheel format, pip is actually doing it right; PEP 427 is wrong (and should be fixed).

[uranusjr]

Cross-linking python/peps#1824.

[ivirshup]

For backwards compatibility, shouldn't wheels which are built following the current PEP-427 mangling and 440 version specs be installable? My feeling is that a wheel made following current PEP specs should be allowed, regardless of whether the spec might be amended. Once amended, isn't some period of backward compatibility still expected?

My practical concern is: I'd like to keep pip up to date on my projects CI. Using flit for our build system requires that we pin pip to an older version until this is resolved.

[uranusjr]

The problem is, if both + and - are escaped into _, pip cannot know if 0.1_2 came from 0.1+2 or 0.1-2 (both are valid versions) and can't even recognise the file is a match or not, let along allowing it.

[ivirshup]

The version should still be resolvable from the name in the metadata, just not the filename, right? The metadata wouldn't have been mangled (at least that's my understanding).

Here's the last few lines of what I see if I try pip installing my project using flit as the build tool:

  WARNING: Built wheel for scanpy is invalid: Wheel has unexpected file name: expected '1.7.0rc2.dev33+gf953fbc8', got '1.7.0rc2.dev33-gf953fbc8'
Failed to build scanpy
ERROR: Could not build wheels for scanpy which use PEP 517 and cannot be installed directly

It looks to me like the build is failing to install because the metadata doesn't match the filename, not that pip is unable to figure out the version string.

[uranusjr]

Sure, but how does pip find that file in the first place? Download all files that look like it and check one by one? That decision would probably not be too popular.

[takluyver]

In pypa/flit#383, people are running into this error in a context where pip doesn't need to find the version: running pip install . (via flit install). Pip gets a specific wheel file, and then refuses to install it because of the name.

I understand that local version identifiers (with a +) can't be uploaded to PyPI anyway. Of course, you could still have one of these packages in a private index or folder - but as people only started reporting this after pip added an explicit check, I suspect that's rare. 🙂

[ivirshup]

Exactly what @takluyver said, there isn't any searching for a package in this use case, as these wheels couldn't be uploaded to PyPI.