Zero install can lead to runaway repo size, doing permanent damage

[conartist6]

Yarn recommends zero-installs. On the zero installs page, runaway repository size is not mentioned as a drawback of zero installs, but this is a condition encountered by some people who have opted into zero installs for repos the meet certain conditions such as having large dependencies and/or numerous dependencies which change regularly. If such a repo opts into zero installs, repo size may become orders of magnitude larger than it would otherwise be, making full repo clones inconvenient or even prohibitively costly, depending on the available network and storage resources.

This problem cannot be truly fixed without eliminating or rewriting the entire history of the repository, both of which are options that cause significant disruptions to development, e.g. by destroying the bases for existing branches.

Does yarn have any responsibility to help people avoid this pitfall or help them recover if they encounter it?

Note: Existing discussion of the topic has occurred on the now-locked #180.

[conartist6]

Post discussion thread (github discussions does not allow comments on the original post)

[conartist6]

The maintainers locked and limited the original issue. Because that action was taken before this discussion was created, github did not add a line to the original issue noting that it is referenced in this discussion, meaning that users that find the issue won't know to come here, and will be effectively gagged from talking about their problem. Could a maintainer add a comment linking the issue to this discussion?

[arcanis]

Done 👍

[conartist6]

Thanks!!

[arcanis]

Note that I've expanded #4839 to include the remediations I could think of (partial clone, sparse checkout, history pruning, lfs), with their pros and cons. I might have missed some 🤔

[PutziSan]

Hi @arcanis, I've noticed that the most recent version of the documentation seems to be missing the section on remediations. Was this an intentional change? Thank you for your time and effort on this project!

[spenserblack]

This problem cannot be truly fixed without eliminating or rewriting the entire history of the repository

I wonder if anyone has had success making the yarn cache a submodule? That way you don't need to rewrite the main repo's history, only the cache repo's history.

A downside is that submodules can be annoying to use, and a user would have to remember to do things like git pull --recurse-submodules instead of git pull. It's kind of less "Plug'n'Play" when you need to remember to manage the submodule.

[arcanis]

It'd be interesting to have a plugin that would automatically manage such a submodule (committing after installs, resolving merge conflicts, etc) 🤔

[spenserblack]

That sounds good! Such a plugin could also possibly have a command like squash-cache, which would switch to an orphan branch for a new history, commit the current state of the cache as a single commit, push up that new branch, and then switch the submodule to the head of that new branch in the main repo.
Basically, something like this:

cd .yarn/cache
# maybe warn if the cache has a dirty history before doing this
git checkout --orphan=cache-abc123 # some unique hash
git commit -m "squash cache history from {ref}" # AFAIK new orphan branches always stage the current files
git push -u orign cache-abc123
cd ../..
git add .yarn/cache
git commit -m "yarn: switch cache to squashed history"

It would then be up to the owner of the submodule repo to change the default branch to cache-abc123 and delete the old branch.
I was originally going to suggest force-pushing instead of an orphan branch, but I think the safety of avoiding a force-push outweighs the annoyance of manually deleting the old branch from the remote.

[devinrhode2]

Some git config settings can make submodules much easier to manage:

[submodule]
  recurse = true
[fetch]
  recurseSubmodules = on-demand
[push]
  recurseSubmodules = on-demand

[This get's a little rambly, wishy-washy, and is not well-edited]

Submodules have been receiving a lot of attention from git core over the past few years, and there's still many more improvements to come. They've gotten a bad wrap, but have very unique characteristics.

I do have quite extensive experience with submodules.

In another thread, I mentioned having an npm caching proxy that can't have it's cache cleared (a sticky "Perma-cache").

Making the .yarn/cache folder a git submodule is not hard.

Add packages to it, easy. Auto-commit, just one more step. Auto-commit in parent: highly opinionated approach I would take, but easy to write the code for that, just stage everything to be committed.

But, in reality, it's just too clunky for what we need here. We want a perma cache, using a git submodule doesn't seem like the best implementation. We can STILL use git, yes, but a git submodule, I'm less sure about. I think a sibling directory might be nice. Maybe we could hide the cache from git with a simple .gitignore entry:

.yarn/cache

Little does git know, when yarn adds and removes packages from that directory, we could also cd into that folder and run git commits, pushes, and resets. Hrmm.. nvm... sounds just like a normal git submodule - except if using git submodules, git can automatically handle most of those updates...

Although, sometimes we DON'T want to update those dependencies when switching around to different branches. Developers would need to pass in a --no-recurse-submodules flag.

I wonder if there could be a lazily-updated .yarn/cache.

Git can keep track of what is on disk, and yarn can ask git, "What is currently on disk?", but yarn doesn't bother updating what packages are in .yarn/cache or in node_modules until some process actually calls require or import for that package. [Optionally, yarn could have a setting to install all packages up-front when running any yarn start or yarn run dev or yarn test command.]

[conartist6]

Since nobody is saying it, one great option to avoid this problem (assuming you haven't already created it) is simply not checking in the cache! You could do this either by adding .yarn/cache (the local cache) to your project's .gitignore, or by configuring enableGlobalCache. Then instead of needing to store every version of every dependency you've ever used for every repo, you can use a centralized source of truth for the definition of packages and their historical contents! This radical technology is battle-tested, keeps your repos efficient, and still allows you to work offline so long as your cache is populated.

The only significant drawback of this approach is "the left-pad problem". It is possible for npm authors to delete their packages. Few people do this, but if it happens you may be left without a copy of package you depend on. For this reason it is advisable for the most risk-averse developers like major corporations to configure their package registry to be a caching proxy of the main npm package registry.

Doing this will (may) require you to run yarn install after using your VCS to check out a version which has different dependencies.

[arcanis]

I mean, it's basically the opposite of the pros: you have to remember to run yarn install each time you switch branches, which tends to happen far more than clones in my experience, and you not only suffer from "left-pad"-style issue, but also from connectivity ones (and the npm registry is far from having 100% uptime). And of course, if you use private registries, you also need to make sure the auth is properly configured for everyone (whereas if the packages are in the repository, there is no special setup needed).

[conartist6]

There are also public mirrors of the npm registry in cases where the npm registry is down, and if you've got what you need in the global package cache you shouldn't need the npm registry either, so at least you wouldn't lose access to something that was already working.

Yeah you need to run yarn install. My attitude towards that is you give something, and you get something. That's also the case with zero-install, which over the long run seems to pressure you towards using sparse checkouts.

[devinrhode2]

Need to re-run yarn install when switching branches

There are git hooks that can run yarn install on post-checkout, they are quite nice.

But, sometimes you change branches, and may not want to re-run yarn install.
A git hook could simply privately keep tabs on which yarn.lock generated your current node_modules folder. (Just store a hash of yarn.lock contents in a secret file, .node_modules_key.

Then, whenever a command like yarn start or yarn run dev or yarn test runs, yarn could simply check if the existing hash of yarn.lock matches what's in .node_modules_key. And, then it may or may not decide to re-run yarn install. Laziness at its best.

I think yarn could integrate this behavior by default.

[devinrhode2]

My gut instinct, for "left-pad" and reliability issues, would be to have a "more robust" global yarn cache on the operating system. But, this is wrong. After 5 years, things will have been cleared from the cache, if you want them, you may not be able to find them. I think the proper solution would be to make immutable package registries, some sort of blockchain inspired thing. Sadly, I'm not sure npm/microsoft could or would totally remove the ability for them to unpublish/delete packages. But, they could at least setup terms and systems where nobody outside of microsoft proper can unpublish/delete packages. Archive versions, sure. Mark them deprecated, sure. But once something is published, and somebody installs it, then it's locked in, and will be there forever.

[conartist6]

An alternative that I'm personally working on making viable is storing your transpiled code in git. This is one step short of zero-install, and while you could do both, I'd recommend against it. While checking in transpiled code may make your repo some constant size factor larger than it would have been, it will not create exponential size runaway like zero-install can, and it still has significant benefits:

• It allows packages to be decentralized and forkable: all you need to do is specify the version of a package as its repo/commitish, e.g. "some-package": "github:me/some-package".
• It ensures that you are protected from tool rot: e.g. the tendency of an environment to evolve and possibly make build tools not work at all or behave differently than they once did.
• It makes it trivial for your test framework to run against the real code.
• It allows you to see version-to-version diffs of the actual shippable code, making it easy to analyze a package for security concerns or possible breakages. It also allows you to see per-commit diffs in the transpiled output, allowing you to easily see the effects caused by changes in the configuration of your transpiler.

The biggest drawback to the approach is that it requires some tooling that doesn't exist yet in order to avoid making a mess of your repo. I am working on building that tooling as macrome.

[devinrhode2]

I'm a fan of this. When it comes to security, the code we're shipping is what is of utmost importance.

Also, it's faster to go back in time and see how something worked. Or, see how your co-workers PR works, etc.

[bryanph]

Does committing uncompressed files behave different over time? Right now a small change in a package results in a new file but when committing package files uncompressed this is not the case? Also git does some compression out of the box perhaps that's enough? Perhaps the additional storage at the start becomes worth it over time by having smaller diffs on package upgrades?

I guess the main problem in this case would be minified files which I don't think git can diff because it compares line-by-line if I remember correctly. Although it's probably smarter than that I really don't know 😁

Or another way might be to make git aware of how to diff the archives: https://stackoverflow.com/a/8001900/1918818

[arcanis]

iirc, from @andreialecu experiments, git was able to diff uncompressed binary archives when optimizing the repository. So while it was sometimes more expensive, it was generally better to use uncompressed archives with zero-install.

It would be nice to add a test in the codebase to exhibit this behavior more clearly, and mention it in the documentation.

[devinrhode2]

While fiber optic connections are on the rise, upload speed over cable connections is going to be a bottle neck for many years to come.

By uploading our own copies of package zip files, via git, we've committed a horrendous sin when it comes to optimal networking. Not only is that upload going to be 100x slower than the download, it causes everyone else to need to download the pseudo-unique copy we uploaded.

I think having a caching npm proxy that never clears the cache would be a fair solution.