CI fails with nixpkgs after pip 20.3 update

[infinisil]

After bisecting this, I found that CI started failing with the pip 20.2.4 -> 20.3 update in NixOS/nixpkgs@b0e3728. The error is in regards to git dependencies:

Collecting alembic@ git+https://github.com/sqlalchemy/alembic.git@rel_1_3_1
  Cloning https://github.com/sqlalchemy/alembic.git (to revision rel_1_3_1) to /build/pip-install-i48hvlvt/alembic_3450e6726d6b44b6ab41a7e579ae0c67
  ERROR: Error [Errno 2] No such file or directory: 'git' while executing command git clone -q https://github.com/sqlalchemy/alembic.git /build/pip-install-i48hvlvt/alembic_3450e6726d6b44b6ab41a7e579ae0c67
ERROR: Cannot find command 'git' - do you have 'git' installed and in your PATH?
builder for '/nix/store/df9ciykjrpyym58kc88vlcdqdjaq6sii-python3.8-git-deps-0.1.0.drv' failed with exit code 1

For some reason pip doesn't detect the passed dependency anymore and tries to fetch it on its own. This error is reproducible with

$ nix-build tests -A git-deps -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/master.tar.gz

[adisbladis]

cc @FRidh

[adisbladis]

I'm thinking to add an override pinning pip at 20.2.4 as an intermediate workaround and cut a new release of poetry2nix.

[adisbladis]

This issue has been worked around in #248, but I'm going to keep this issue open until we have a proper fix.

[cpcloud]

Just to pile on here, this pin causes building wheels to fail in some cases due to the generation of manylinux_x_y wheels, which require pip >= 20.3. The workaround is

poetry2nix/overrides.nix

Lines 1875 to 1877 in 3000009

	preInstall = lib.optionalString stdenv.isLinux ''
	mv ./dist/*.whl $(echo ./dist/*.whl | sed s/'manylinux_[0-9]*_[0-9]*'/'manylinux1'/)
	'';

.

[cpcloud]

Piling on again, I think this is also preventing using poetry2nix and Python 3.10 for a large number of packages. msgpack for one will not build with Python 3.10, even with the workaround it seems.

[cpcloud]

I can confirm that removing the workaround allows msgpack to build successfully with Python 3.10 using poetry2nix.

[cpcloud]

Is this actually still a problem? The OP's repro step no longer fails.

[cpcloud]

Ok, it does seem to reproduce when I run nix-build tests. Does this indicate a need to bump the minimum version of nixpkgs that poetry2nix is using?

[cpcloud]

I've started bisecting this and hope to have at least a first breaking pip commit hash shortly.

[cpcloud]

I wasn't able to find an exact commit after multiple hours of debugging, because the resolver itself is hidden behind flags whose name changes between commits of pip, and whose functionality has no clear point of origin, so I had a lot of trouble finding a commit in pip where 1) the new resolver wasn't cloning on every run, and 2) where the new resolver was enabled.

After hunting around the pip issue tracker it seems that the behavior of unconditional cloning was indeed introduced as part of the new (2020) resolver.

See pypa/pip#9244 and the underlying issue pypa/pip#5780.

All that said, I believe we can handle git dependencies that same way we are handling path dependencies, since the nix derivation for a git dependency will already exist by the time a dependent's pyproject.toml processing occurs. I'll have a PR up in bit.