Do wheel installs from VCS/directories using ephemeral caching #4764

pradyunsg · 2017-10-04T13:03:32Z

Closes #4501
Closes #4562
Closes #4714
Closes #536

Quoting from #4501:

Follow up from gh-4144

To allow build system abstractions, we want installation to go through wheels in more cases. In particular, installing packages from a local directory or a VCS URL currently runs 'setup.py install'. The aim of this PR is to have it build a wheel, which is stored in an ephemeral cache directory, used for installation, and then discarded.

We can't cache it permanently based on the path/URL, because the code there might change, but our cache wouldn't be invalidated.

This is implemented by reusing the current WheelCache and wrapping what is basically 2 instances of it in a wrapper that implements everything a Cache should, with an extra helper for direct access to the ephemeral directory when caching wheels.

I'm building on top of some awesome work by @takluyver, @alex and @xoviat.

Follow up from pypagh-4144 To allow build system abstractions, we want installation to go through wheels in more cases. In particular, installing packages from a local directory or a VCS URL currently runs 'setup.py install'. The aim of this PR is to have it build a wheel, which is stored in an ephemeral cache directory, used for installation, and then discarded. We can't cache it permanently based on the path/URL, because the code there might change, but our cache wouldn't be invalidated.

When there's a previous build directory, no_clean is set

pradyunsg · 2017-10-04T13:09:06Z

src/pip/_internal/cache.py

@@ -102,13 +103,18 @@ def _link_for_candidate(self, link, candidate):

        return index.Link(path_to_url(path))

+    def cleanup(self):


I should note that @xoviat had proposed in #4714 to make Caches into context managers, to avoid using this cleanup() pattern.

I'm still confused about how I feel about that. FWIW, we can do that in a later PR. I have some other ideas too but I'd prefer to experiment with that after this PR is done.

@rhettinger What do you think about using context managers for cleanup? You're the person that I got this from.

Just to be clear, I definitely think using a context manager is a good idea. But I agree with @pradyunsg that it's something we should defer until a followup PR. Let's get the functionality landed then address the cleanup.

I agree with that; I was just wondering what others thought in principle.

pradyunsg · 2017-10-04T13:18:16Z

For some reason, my newer commits aren't showing up on GitHub's UI. :/

pradyunsg · 2017-11-14T06:09:26Z

@pypa/pip-committers This PR does not depend on the current PEP 518 implementation; it's something we should probably be doing anyway.

Can we merge this?

pfmoore · 2017-11-14T08:44:01Z

+1 from me

BrownTruck · 2017-11-16T23:45:01Z

Hello!

I am an automated bot and I have noticed that this pull request is not currently able to be merged. If you are able to either merge the master branch into this pull request or rebase this pull request against master then it will eligible for code review and hopefully merging!

pradyunsg · 2017-12-01T13:50:25Z

Okay. This should make this PR mergeable.

@pypa/pip-committers Do we want to go ahead with this, regardless of PEP 517/518?

pfmoore · 2017-12-01T13:53:31Z

+1 from me. At this point, I'm not sure if we'll need to do a pip 10 release without PEP 517/518, but I'd rather this be merged just in case we decide to.

pradyunsg · 2017-12-01T13:58:16Z

pip 10 release without PEP 517/518

@pfmoore You think it's a good idea to discuss this on #4802?

pradyunsg · 2018-01-23T20:52:11Z

@pypa/pip-committers Ping! :)

I still think this change can go in on it's own. Is there something we want to do before merging this?

Aside: When merging this PR, please use plain merge instead of squash/rebase merges since I have other branches sitting on top of this one.

pfmoore · 2018-01-23T22:08:37Z

OK, merged - hope I got the right type of merge for you.

pradyunsg · 2018-01-24T11:37:02Z

Yeps. Thanks! ^>^

lock · 2019-06-02T09:05:34Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

takluyver and others added 23 commits July 5, 2017 20:04

Add news file

a32ebea

Fix list comp

fe5e9cc

Code style fixes

30109c3

Allow temp files for another test

ee67867

When there's a previous build directory, no_clean is set

Fix cleanup on 'pip install' with no args

46c2c54

Cleanup wheel cache after 'pip freeze'

9fdcfaf

Expect temp files in another test

9109322

Update test of install output

ad242d1

Expect temp files on another test

a51843a

Update another install output test

73440ed

Update test to check for dist-info

5610636

Remove string formatting operation

1c03c1b

Install 'wheel' for test that now requires it

ad50ac9

Fix some changes that I screwed up in the rebase

d045cc2

Switch to using the existing tempdir abstraction

01d226b

flake8

d5e3082

fix

1997456

not a thing anymore

328f607

these are probably needed

8ef8400

flake8

cb56bb3

fix

5f88252

Merge master

ebf06ce

pradyunsg self-assigned this Oct 4, 2017

pradyunsg requested a review from a team October 4, 2017 13:03

pradyunsg added type: enhancement Improvements to functionality C: cache Dealing with cache and files in it labels Oct 4, 2017

pradyunsg commented Oct 4, 2017

View reviewed changes

pradyunsg mentioned this pull request Oct 4, 2017

try to build wheels for directories/vcs #4714

Closed

pradyunsg mentioned this pull request Oct 20, 2017

Remove Circular dependency between Resolver and Preparer #4636

Merged

pfmoore mentioned this pull request Oct 21, 2017

Toward PEP 518 #4802

Closed

Merge branch 'master' into cache/ephem-wheel-cache

d4e2d7e

pradyunsg mentioned this pull request Oct 24, 2017

pip should reinstall . #536

Closed

pradyunsg added 2 commits October 24, 2017 15:44

Merge branch 'master' into cache/ephem-wheel-cache

e36bb6d

🎨

a39f36a

BrownTruck added the needs rebase or merge PR has conflicts with current master label Nov 16, 2017

Merge branch 'master' into cache/ephem-wheel-cache

01d97e7

pypa-bot removed the needs rebase or merge PR has conflicts with current master label Nov 17, 2017

pradyunsg mentioned this pull request Dec 1, 2017

Feature/pipignore #4900

Closed

pradyunsg added 2 commits December 1, 2017 19:16

Merge branch 'master' into cache/ephem-wheel-cache

cd14240

Merge branch 'master' into cache/ephem-wheel-cache

9b6f1d7

xavfernandez mentioned this pull request Dec 9, 2017

Underscore in package name confuses version identification, prevents hash-based upgrade #3212

Closed

Merge branch 'master' into cache/ephem-wheel-cache

4c14739

pfmoore merged commit 81fb515 into pypa:master Jan 23, 2018

pradyunsg deleted the cache/ephem-wheel-cache branch January 24, 2018 11:37

zougloub mentioned this pull request Mar 1, 2018

installation is very long spinalcordtoolbox/spinalcordtoolbox#1549

Closed

pradyunsg removed the !release blocker Hold a release until this is resolved label Jul 15, 2018

lock bot added the auto-locked Outdated issues that have been locked by automation label Jun 2, 2019

lock bot locked as resolved and limited conversation to collaborators Jun 2, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Do wheel installs from VCS/directories using ephemeral caching #4764

Do wheel installs from VCS/directories using ephemeral caching #4764

pradyunsg commented Oct 4, 2017 •

edited

Loading

pradyunsg Oct 4, 2017 •

edited

Loading

ghost Oct 4, 2017

pfmoore Oct 4, 2017

ghost Oct 4, 2017

pradyunsg commented Oct 4, 2017

pradyunsg commented Nov 14, 2017

pfmoore commented Nov 14, 2017

BrownTruck commented Nov 16, 2017

pradyunsg commented Dec 1, 2017

pfmoore commented Dec 1, 2017

pradyunsg commented Dec 1, 2017

pradyunsg commented Jan 23, 2018

pfmoore commented Jan 23, 2018

pradyunsg commented Jan 24, 2018

lock bot commented Jun 2, 2019

		@@ -102,13 +103,18 @@ def _link_for_candidate(self, link, candidate):

		return index.Link(path_to_url(path))

		def cleanup(self):

Do wheel installs from VCS/directories using ephemeral caching #4764

Do wheel installs from VCS/directories using ephemeral caching #4764

Conversation

pradyunsg commented Oct 4, 2017 • edited Loading

pradyunsg Oct 4, 2017 • edited Loading

Choose a reason for hiding this comment

ghost Oct 4, 2017

Choose a reason for hiding this comment

pfmoore Oct 4, 2017

Choose a reason for hiding this comment

ghost Oct 4, 2017

Choose a reason for hiding this comment

pradyunsg commented Oct 4, 2017

pradyunsg commented Nov 14, 2017

pfmoore commented Nov 14, 2017

BrownTruck commented Nov 16, 2017

pradyunsg commented Dec 1, 2017

pfmoore commented Dec 1, 2017

pradyunsg commented Dec 1, 2017

pradyunsg commented Jan 23, 2018

pfmoore commented Jan 23, 2018

pradyunsg commented Jan 24, 2018

lock bot commented Jun 2, 2019

pradyunsg commented Oct 4, 2017 •

edited

Loading

pradyunsg Oct 4, 2017 •

edited

Loading