pip install --find-links not expanding "~"

[j0057]

Environment

• pip version: >19.0.3
• Python version: 3.7.3
• OS: ArchLinux

At the moment I have pip 19.0.3 installed in my user-site directory. I noticed a change in behavior when using a newer pip (19.2.1) while experimenting with PYTHONUSERBASE as an alternative to virtualenvs.

Description
The --find-links option used to expand tildes (~) into the user's $HOME, but it no longer does so.

Expected behavior
The --find-links option should expand tildes like it used to.

How to Reproduce

I was able to bisect it to commit 7d08bb3 as the first commit that no longer expands tildes: (All done from the root of the pip repo)

1. Use the current master version of pip to download a wheel such as pytz: PYTHONPATH=src python -m pip download --destination-dir='~/temp' pytz
2. Start bisecting: git bisect start HEAD 19.0.3
3. Repeatedly install pytz into a custom PYTHONUSERBASE: PYTHONPATH=src PYTHONUSERBASE=/tmp/x git bisect run sh -c "rm -rf /tmp/x; python -m pip --isolated --no-cache-dir --disable-pip-version-check install --no-index --find-links='~/temp' --user pytz"

(Clean everything up: rm -rf /tmp/x ~/temp)

Output

jjm@muon pip master %= $ PYTHONPATH=src python -m pip download --destination-dir='~/temp' pytz
Collecting pytz
  Downloading https://files.pythonhosted.org/packages/3d/73/fe30c2daaaa0713420d0382b16fbb761409f532c56bdcc514bf7b6262bb6/pytz-2019.1-py2.py3-none-any.whl (510kB)

  Saved /home/jjm/temp/pytz-2019.1-py2.py3-none-any.whl
Successfully downloaded pytz

jjm@muon pip master %= $ git bisect start HEAD 19.0.3
Bisecting: 251 revisions left to test after this (roughly 8 steps)
[f44344f1220bce5cd942641f832cb407ed477f06] Merge pull request #6515 from johnthagen/svn-interactive-final

jjm@muon pip (f44344f1...) %|BISECTING $ PYTHONPATH=src PYTHONUSERBASE=/tmp/x git bisect run sh -c "rm -rf /tmp/x; python -m pip --isolated --no-cache-dir --disable-pip-version-check install --no-index --find-links='~/temp' --user pytz"
running sh -c rm -rf /tmp/x; python -m pip --isolated --no-cache-dir --disable-pip-version-check install --no-index --find-links='~/temp' --user pytz
Looking in links: /home/jjm/temp
Collecting pytz
Installing collected packages: pytz
Successfully installed pytz-2019.1
Bisecting: 126 revisions left to test after this (roughly 7 steps)
[edbd0822dc50f49ef92dc7bee27d5728ed3babb7] Rename InstallRequirement.setup_py -> setup_py_path
running sh -c rm -rf /tmp/x; python -m pip --isolated --no-cache-dir --disable-pip-version-check install --no-index --find-links='~/temp' --user pytz
Looking in links: ~/temp
Collecting pytz
  WARNING: Url '~/temp' is ignored. It is either a non-existing path or lacks a specific scheme.
  ERROR: Could not find a version that satisfies the requirement pytz (from versions: none)
ERROR: No matching distribution found for pytz
Bisecting: 62 revisions left to test after this (roughly 6 steps)
[3f8df2487597d47279b167f29d93622e5632c044] Merge pull request #6600 from lkollar/consolidate-path-opt-parsing
running sh -c rm -rf /tmp/x; python -m pip --isolated --no-cache-dir --disable-pip-version-check install --no-index --find-links='~/temp' --user pytz
Looking in links: /home/jjm/temp
Collecting pytz
Installing collected packages: pytz
Successfully installed pytz-2019.1
Bisecting: 30 revisions left to test after this (roughly 5 steps)
[c3c61b5c5198b702aa506cc72c3cdf78974b88b7] Merge pull request #6624 from cjerdonek/add-search-scope
running sh -c rm -rf /tmp/x; python -m pip --isolated --no-cache-dir --disable-pip-version-check install --no-index --find-links='~/temp' --user pytz
Looking in links: /home/jjm/temp
Collecting pytz
Installing collected packages: pytz
Successfully installed pytz-2019.1
Bisecting: 15 revisions left to test after this (roughly 4 steps)
[a447be47858f980b9b022891ab4d45aebd0ac967] Update PackageFinder to support "yanked" files (PEP 592).
running sh -c rm -rf /tmp/x; python -m pip --isolated --no-cache-dir --disable-pip-version-check install --no-index --find-links='~/temp' --user pytz
Looking in links: ~/temp
Collecting pytz
  WARNING: Url '~/temp' is ignored. It is either a non-existing path or lacks a specific scheme.
  ERROR: Could not find a version that satisfies the requirement pytz (from versions: none)
ERROR: No matching distribution found for pytz
Bisecting: 7 revisions left to test after this (roughly 3 steps)
[36430e2a43b8e820af60b0eb43195fd9f323e5f2] Merge pull request #6639 from sbidoul/imp_looks_like_hash-sbi
running sh -c rm -rf /tmp/x; python -m pip --isolated --no-cache-dir --disable-pip-version-check install --no-index --find-links='~/temp' --user pytz
Looking in links: ~/temp
Collecting pytz
  WARNING: Url '~/temp' is ignored. It is either a non-existing path or lacks a specific scheme.
  ERROR: Could not find a version that satisfies the requirement pytz (from versions: none)
ERROR: No matching distribution found for pytz
Bisecting: 3 revisions left to test after this (roughly 2 steps)
[a38a0eacd4650a7d9af4e6831a2c0aed0b6a0329] Merge pull request #6631 from cjerdonek/finder-search-scope
running sh -c rm -rf /tmp/x; python -m pip --isolated --no-cache-dir --disable-pip-version-check install --no-index --find-links='~/temp' --user pytz
Looking in links: ~/temp
Collecting pytz
  WARNING: Url '~/temp' is ignored. It is either a non-existing path or lacks a specific scheme.
  ERROR: Could not find a version that satisfies the requirement pytz (from versions: none)
ERROR: No matching distribution found for pytz
Bisecting: 0 revisions left to test after this (roughly 1 step)
[cab7316e2b6f994d95527c0577c14f6ae2164126] Change PackageFinder.create() to accept a SearchScope object.
running sh -c rm -rf /tmp/x; python -m pip --isolated --no-cache-dir --disable-pip-version-check install --no-index --find-links='~/temp' --user pytz
Looking in links: ~/temp
Collecting pytz
  WARNING: Url '~/temp' is ignored. It is either a non-existing path or lacks a specific scheme.
  ERROR: Could not find a version that satisfies the requirement pytz (from versions: none)
ERROR: No matching distribution found for pytz
Bisecting: 0 revisions left to test after this (roughly 0 steps)
[761b2064f42e6904bba189d89b29100f7c03ca1f] Remove models/search_scope.py's dependency on download.py.
running sh -c rm -rf /tmp/x; python -m pip --isolated --no-cache-dir --disable-pip-version-check install --no-index --find-links='~/temp' --user pytz
Looking in links: /home/jjm/temp
Collecting pytz
Installing collected packages: pytz
Successfully installed pytz-2019.1
cab7316e2b6f994d95527c0577c14f6ae2164126 is the first bad commit
commit cab7316e2b6f994d95527c0577c14f6ae2164126
Author: Chris Jerdonek <[email protected]>
Date:   Fri Jun 21 09:10:03 2019 -0700

    Change PackageFinder.create() to accept a SearchScope object.

 src/pip/_internal/cli/base_command.py | 16 ++++------------
 src/pip/_internal/cli/cmdoptions.py   | 26 ++++++++++++++++++++++++++
 src/pip/_internal/commands/list.py    | 15 ++++++---------
 src/pip/_internal/commands/wheel.py   |  5 -----
 src/pip/_internal/index.py            | 10 ++--------
 src/pip/_internal/utils/outdated.py   |  6 ++++--
 tests/lib/__init__.py                 |  9 +++++++--
 tests/unit/test_build_env.py          |  4 +++-
 tests/unit/test_cmdoptions.py         | 31 ++++++++++++++++++++++++++++++-
 tests/unit/test_index.py              | 11 +++++++++--
 tests/unit/test_unit_outdated.py      |  4 ++--
 11 files changed, 93 insertions(+), 44 deletions(-)
bisect run success

Bisect log

# bad: [f76014efac796c0656ad61bd2bbcc23b67d66495] Add global options and no user config args to make_setuptools_shim_args (#6706)
# good: [a8de4eb49a45035aefd9a122b6d1d67915663d5f] Generate NEWS
git bisect start 'HEAD' '19.0.3'
# good: [f44344f1220bce5cd942641f832cb407ed477f06] Merge pull request #6515 from johnthagen/svn-interactive-final
git bisect good f44344f1220bce5cd942641f832cb407ed477f06
# bad: [edbd0822dc50f49ef92dc7bee27d5728ed3babb7] Rename InstallRequirement.setup_py -> setup_py_path
git bisect bad edbd0822dc50f49ef92dc7bee27d5728ed3babb7
# good: [3f8df2487597d47279b167f29d93622e5632c044] Merge pull request #6600 from lkollar/consolidate-path-opt-parsing
git bisect good 3f8df2487597d47279b167f29d93622e5632c044
# good: [c3c61b5c5198b702aa506cc72c3cdf78974b88b7] Merge pull request #6624 from cjerdonek/add-search-scope
git bisect good c3c61b5c5198b702aa506cc72c3cdf78974b88b7
# bad: [a447be47858f980b9b022891ab4d45aebd0ac967] Update PackageFinder to support "yanked" files (PEP 592).
git bisect bad a447be47858f980b9b022891ab4d45aebd0ac967
# bad: [36430e2a43b8e820af60b0eb43195fd9f323e5f2] Merge pull request #6639 from sbidoul/imp_looks_like_hash-sbi
git bisect bad 36430e2a43b8e820af60b0eb43195fd9f323e5f2
# bad: [a38a0eacd4650a7d9af4e6831a2c0aed0b6a0329] Merge pull request #6631 from cjerdonek/finder-search-scope
git bisect bad a38a0eacd4650a7d9af4e6831a2c0aed0b6a0329
# bad: [cab7316e2b6f994d95527c0577c14f6ae2164126] Change PackageFinder.create() to accept a SearchScope object.
git bisect bad cab7316e2b6f994d95527c0577c14f6ae2164126
# good: [761b2064f42e6904bba189d89b29100f7c03ca1f] Remove models/search_scope.py's dependency on download.py.
git bisect good 761b2064f42e6904bba189d89b29100f7c03ca1f
# first bad commit: [cab7316e2b6f994d95527c0577c14f6ae2164126] Change PackageFinder.create() to accept a SearchScope object.

[cjerdonek]

I was able to bisect it to commit 7d08bb3 as the first commit that no longer expands tildes: (All done from the root of the pip repo)

I think you mean commit cab7316?

It looks like the error is that this line should have been SearchScope.create() rather than SearchScope(): cab7316#diff-863212c0ea2f09c6463884971d77230dR371

[j0057]

Yes, you're right, I did mean cab7316.

I tested the fix you propose, that indeed fixes it. I'll send a PR for it!

[cjerdonek]

Thanks. In your PR you should add another unit test of that function testing that e.g. normalize_path is called on such values.

[cjerdonek]

I posted PR #6827 to fix this, with tests, and added it to the 19.2 milestone.

[pradyunsg]

Released pip 19.2.2 containing a fix for this issue. :)