Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce cache clobbering within cache_file_entry() #32042

Closed
staticfloat opened this issue May 15, 2019 · 2 comments
Closed

Reduce cache clobbering within cache_file_entry() #32042

staticfloat opened this issue May 15, 2019 · 2 comments

Comments

@staticfloat
Copy link
Sponsor Member

As things stand, the precompile cache is clobbered very regularly; A project with Foo.jl installed at version 1.2 and a project with Foo.jl dev'ed to ~/.julia/dev also at version 1.2 will clobber eachother's cached .ji files. This is particularly frustrating when you have two branches of a package checked out in two different projects that are technically at the same version; running code in both of those projects at the same time virtually ensures that you will be precompiling every time you load code.

I propose that we introduce a path parameter, so that we call cache_file_entry(pkg, path), and hash path into pkg.uuid to generate the package slug. There are only two places where we invoke cache_file_entry() and in both places we have path information readily available. While this would potentially cause a multiplication in the number of precompiled images, I think it would be worthwhile to err on the side of saving time rather than space.

@tkf
Copy link
Member

tkf commented May 16, 2019

How about the cache files for package Bar.jl depending on Foo.jl? Same version of Bar.jl may be used with different versions of Foo.jl including dev'ed version. I don't think it is enough to rely on package identity and path if you want to eliminate unnecessary precompilation. If you want to get rid of unnecessary precompilation (and save space), I think you need to generate cache paths based on something like hash tree. It would be pretty challenging considering the dynamic nature of package loading in Julia (see, e.g., #27418 (comment)).

If you want to error on the side of saving time, I think a better option would be to use per-project cache path. (See my jlm CLI that does that https://discourse.julialang.org/t/22849)

@staticfloat
Copy link
Sponsor Member Author

Thanks @tkf, I'm going to close this in favor of #27418, which has lots of great discussion already. I'll just go there and add my voice in with the crowd.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants