Vendor in pip==22.0.4

[matteius]

This PR is currently isolated to upgrading vendor'd pip==22.0.4 but there seems is a bug with how we or something internal use specifiers and some follow-up changes required, to separate those out: #4969

• pip 22.x drops support for python 3.6 which means we will have to drop python 3.6 support upon finalizing this, so if there are other bug fixes and another release or two of pipenv to support python 3.6 those should come first.

The issue

The newer pip version has some fixes to dependency resolution that we rely on in pipenv.

So for the example Pipfile I am trying to solve the Infinity lock, and now locking fails with logical error:

Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✘ Locking Failed! 

ERROR:pip.subprocessor:[present-diagnostic] pip subprocess to install build dependencies exited with 1
[ResolutionFailure]:   File "/home/matteius/pipenv/pipenv/resolver.py", line 743, in _main
[ResolutionFailure]:       resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages, dev)
[ResolutionFailure]:   File "/home/matteius/pipenv/pipenv/resolver.py", line 704, in resolve_packages
[ResolutionFailure]:       results, resolver = resolve(
[ResolutionFailure]:   File "/home/matteius/pipenv/pipenv/resolver.py", line 685, in resolve
[ResolutionFailure]:       return resolve_deps(
[ResolutionFailure]:   File "/home/matteius/pipenv/pipenv/utils.py", line 1378, in resolve_deps
[ResolutionFailure]:       results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
[ResolutionFailure]:   File "/home/matteius/pipenv/pipenv/utils.py", line 1107, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "/home/matteius/pipenv/pipenv/utils.py", line 883, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: pip subprocess to install build dependencies exited with 1

The fix

How does this pull request fix your problem? Did you consider any alternatives? Why is this the best solution, in your opinion?

Upgrading vendor'd pip is the path forward for a number of resolver bugs.

The checklist

• Associated issue
• A news fragment in the news/ directory to describe this fix with the extension .bugfix, .feature, .behavior, .doc. .vendor. or .trivial (this will appear in the release changelog). Use semantic line breaks and name the file after the issue number or the PR #.

[matteius]

I've updated the pip patches and got the vendoring part of the build passing. I have a much better grasp of what is going on there now.

[mungojam]

I'd appreciate if you can fix #5008 before dropping 3.6 support. It is our route to later versions of python.

[matteius]

For some reason the build is failing on the linting step because something is causing it to drop the pyproject.toml file which has the black configuration for line length. Very odd: https://github.com/pypa/pipenv/runs/6063647329?check_suite_focus=true#step:7:789

[matteius]

@oz123 There are two primary changes that were required to fix the build. Well its still running but I have good indicators that this run will be fixed. Here is the intel:

The first issue was to drop a patch we had for pip_shims that was causing it to get confused by two different install locations of packaging -- this one was confusing and causing a bulk of the test failures, it seems a bunch of imports are mixed between using pipenv.vendor.packaging and using packaging directly which who knows where that one comes from because of the odd monkey patching logic in parts of the code, plus additionally the vendor'd pip has its own packakging requirement that it references. I went through several iterations of trying to be consistent without success before realizing dropping this patch solved the problem and that solved for all but one of the tests.

Which brings me to the second issue: for some reason the old pip vendoring didn't mind my prior refactor to collect_hashes but with the new pip vendoring there was an issue with the test test_urls_work because the hash it found on generation of the lock file didn't match what it determined to use during the install. Changing back this one part of the refactor to collect_hashes solved for that, and I followed up with a manual test of pytorch hash collection which was the reason I had done that refactor work to begin with, and thankfully the hash collection works still for that example:

PS C:\Users\matte\Projects\pipenv-triage\pipenv-5022e> cat .\Pipfile.lock
{
    "_meta": {
        "hash": {
            "sha256": "8981cd0d0e70e333d8ebc658193de2510698d4a01b1fa79b5e3462299ecfbfe8"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.10"
        },
        "sources": [
            {
                "name": "downloadpytorch",
                "url": "https://download.pytorch.org/whl/cu113/",
                "verify_ssl": true
            },
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "torch": {
            "hashes": [
                "sha256:7fd4751bbf39bbb04ec6116c7243ce6528aded4197afcf380537340e1eebd19a",
                "sha256:a68c33657a546131eb9bc44e2a98d2fa704aafae861460b051b82813852ccb44",
                "sha256:b6a799bdb6ee3d914e5e62bddb4276d4a10248c1af4f2d217738e5f9ee27485b",
                "sha256:ddc57495195aa2456e78bfc7d8d3f45dabbb8b7b268b3b5dbed4f0e4db492f33",
                "sha256:e4bb14d953db9aad5bdb945a328410638721d77e3e622d0a8d77063c01daf40b",
                "sha256:e9126b0a5d95704bee40a9d0ef1cbd82d8dc7863e4638a376bef702dfd659370",
                "sha256:e9df65c1fb2d80283b276114878fd38f411b70880e0b406c451d000e6159f451",
                "sha256:f56333470daea3c97078b37607e0035cccf72fc5c36fd84546e1a4b8d9944f2b"
            ],
            "index": "downloadpytorch",
            "version": "==1.11.0+cu113"
        },
        "typing-extensions": {
            "hashes": [
                "sha256:6657594ee297170d19f67d55c05852a874e7eb634f4f753dbd667855e07c1708",
                "sha256:f1c24655a0da0d1b67f07e17a5e6b2a105894e6824b92096378bb3668ef02376"
            ],
            "index": "pypi",
            "version": "==4.2.0"
        }
    },
    "develop": {}
}

[oz123]

Awesome work!