[fast-deps] Caching during dependency resolution

[McSinyx]

Depending on the command, the file may already exist:

1. pip install - wheel cache or HTTP cache
2. pip wheel - wheel download folder, wheel cache, or HTTP cache
3. pip download - download folder, wheel cache, or HTTP cache

In all three cases, I think the wheel cache piece may be handled earlier during resolving, so we wouldn't inadvertently use a lazy wheel for it.

In all three cases, it may be the case that the file we want is already in the HTTP cache. I don't know if range requests bypass the HTTP cache or it does what we want (which would be to return the requested bytes from the file on disk). Maybe it is worth it to see if the request is cached first, and just skip the lazy wheel?

In pip wheel and pip download, the user may have already downloaded some of the files, and we would want to be able to use those rather than lazily downloading the metadata. Note that the logic refactored in #8685 does not account for this.

Originally posted by @chrahunt in #8697 (comment)

[McSinyx]

The first thing I'll try to do is to avoid the lazy wheel from being used if the distribution is available in the wheel cache, and probably the download directory since the two are handled pretty closely in the execution path.

pip's current implementation of HTTP cache doesn't work out of the box for range requests (GH-8701) but it might be possible.

[pradyunsg]

If the distribution is in the cache, I think we can skip the fast-deps logic for it. There's no point doing a "partial" download, for something that's available in a local cache.

[chrahunt]

A few possible approaches for handling the HTTP cache:

1. query the HTTP cache. This adds some more dependencies between internal components, but it would all be in pip itself.
2. if cachecontrol is Range-request aware and returns from_cache=True on responses for files that are already fully cached, then the point at which we'd know to not use fast-deps for the file would be the first request made inside LazyWheel
3. if cachecontrol is Range-request aware and we ignore whether the response is cached, then there's a little bit more overhead than there would be otherwise for metadata but the code stays pretty clean compared to options 1 and 2

The last two assume we make cachecontrol Range-request aware, and I'm not sure what the effort for that would be. I would guess not much if there's agreement on how it should behave. If we go that way then it'd be good to compare what we can do internally vs changes that'd be best made upstream in cachecontrol itself.

[McSinyx]

I just found out that wheel cache is handled by the resolver:

pip/src/pip/_internal/resolution/resolvelib/candidates.py

Lines 265 to 281 in c50307e

	class LinkCandidate(_InstallRequirementBackedCandidate):
	is_editable = False
	def __init__(
	self,
	link, # type: Link
	template, # type: InstallRequirement
	factory, # type: Factory
	name=None, # type: Optional[str]
	version=None, # type: Optional[_BaseVersion]
	):
	# type: (...) -> None
	source_link = link
	cache_entry = factory.get_wheel_cache_entry(link, name)
	if cache_entry is not None:
	logger.debug("Using cached wheel link: %s", cache_entry.link)
	link = cache_entry.link

Download folders will be checked with GH-8804 merged. I will try to see what are the possibilities with HTTP cache.