Cannot install package offline with VCS dependencies, even if they are pre-installed

[alkasm]

Description

Offline installations of a package with a VCS URL dependency fails even if the dependency is already installed. This happens because pip tries to pull the VCS URL, and when it can't it fails.

Recently (not sure when, possibly when the new resolver rolled out), I believe VCS-specified dependencies became "eager by default" in pip, so that they are downloaded even when already installed (to install and check version metadata, etc). This means that invoking pip install with VCS URL dependencies fails offline. To counteract this, we can use --no-deps to not install the dependencies, but then you must forego the version checking that pip does on all the other packages which are currently installed. Alternatively, I get around both problems using --use-deprecated legacy-resolver.

I marked this as a "bug" rather than a "feature request" as the old resolver worked here, but the new resolver does not.

I see at least two workable solutions for this problem, maybe there are other ideas:

• add a flag to not eagerly install vcs-specified dependencies if they're already installed
• add a command to explicitly check against version requirements (e.g. check your environment meets the current packages) (potentially similar to a pip resolve A pip resolve command to convert to transitive == requirements very fast by scanning wheels for static dependency info (WORKING PROTOTYPE!) #7819)

Expected behavior

There should be a way to install a package with VCS dependencies offline while version checking the other requirements.

pip version

21.3

How to Reproduce

I've uploaded a very basic example with a descriptive README here: https://github.com/alkasm/test-vcs-offline

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pradyunsg]

See #5780, which this issue is a specific subset of. Specifically, the intended resolution direction: #5780 (comment)

[sbidoul]

@alkasm I come back here from #10564 (comment).

I'm not entirely sure I follow what you mean with an offline installation of a VCS URL. Could you provide a step-by-step scenario that illustrates your use case ?

[alkasm]

@sbidoul Sure thing. For clarity I mean "offline install of a package which has a VCS URL dependency"

I created a repo with an explanatory README here: https://github.com/alkasm/vcs-url-install-example

edit: lol I just realized in editing the OP that I already created a repo example, but I forgot and this one is a different example. Oh well! Now there's two examples :P

[uranusjr]

I think this is the key statement

offline installations of a package with a VCS URL dependency fails even if the dependency is already installed

[alkasm]

I think this is the key statement

Yep, sorry I think my initial description dived into the details too quickly. I added that as a first line for clarity. Thanks!

[sbidoul]

edit: lol I just realized in editing the OP that I already created a repo example

Oops, sorry I did not see it.

[sbidoul]

Ok, so this use case is definitely not addressed in #10564.

I would think it is theoretically possible to support it, by detecting early that the requested direct URL is already installed and feeding the version of the installed distribution into the resolver.

So I'm going to tentatively reopen, so pip maintainers who are more familiar with the resolver can decide if it is something we should eventually attempt to do or if it is simply not possible or impractical.

[alkasm]

@sbidoul Thanks for re-opening then (should also remove the duplicate tag in that case?). I think I still agree with @pradyunsg that the suggested solution in #5780 (comment) would solve my problem. In particular:

Conversely, never re-install (unless --force-reinstall, of course) if a URL pointing to an archive does not change.

In my case, the URL does not change, so pip should not reinstall it by default. The author of #10564 says that it does implement what is suggested in this comment (in the first comment on the PR), but I haven't verified that; is that not the case?

[sbidoul]

@alkasm I tested #10564 which, in its current implementation, does not address your use case of off-line re-installation. It does not reinstall, as advertised, but it still executes a git clone to prepare metadata, which fails if off-line.

[uranusjr]

I’m conflicted on this particular case. The VCS URL in the example repo is pointing to a variable location (the default branch), and if I remember correctly, there are some people that do want upgrade to automatically happen if the remote branch is updated. I explicitly called out the case in #5780 (comment):

Always re-clone if the URL points to a VCS and pip cannot determine whether the target has changed.

If the example has a pinned-down URL (i.e. points to a commit), I agree network access should not be mandatory.

[alkasm]

I’m conflicted on this particular case. The VCS URL in the example repo is pointing to a variable location (the default branch), and if I remember correctly, there are some people that do want upgrade to automatically happen if the remote branch is updated. I explicitly called out the case in #5780 (comment):

Always re-clone if the URL points to a VCS and pip cannot determine whether the target has changed.

If the example has a pinned-down URL (i.e. points to a commit), I agree network access should not be mandatory.

Hmm but the default behavior of pip for other packages is to not install them if you already have them installed, even if the package is newer, right? So should it be different for VCS URLs? I imagine you could still explicitly request this behavior with --upgrade or similar?

While I'd argue that pip should 'just work' here, my main predicament is I can't even opt into the behavior I need, except by using the legacy resolver, which is deprecated. If I can opt in with a sanctioned flag we'd at least be cookin', maybe not with gas though.

[alkasm]

If the example has a pinned-down URL (i.e. points to a commit), I agree network access should not be mandatory.

@uranusjr note that this isn't really a workable flow for a library; pinning dependencies is something you can only really do for applications. If I have N libraries that depend on some core library (specified via VCS URL), then if I want to bump that core library for any application, I have to update all N libraries that use it (and all the downstream applications using those libraries) in lock step. Otherwise, two will disagree on the specific commit, pip will complain that it can't satisfy the dependencies, and I won't be able to install anything.