Cannot yarn install; looks like some dependencies are missing

[Montoya]

When downloading this repo on my local machine and trying to yarn install in my terminal I get the following errors:

It looks like some dependencies are missing in the package.json?

[ziad-saab]

Tried to yarn install on both Linux and Mac, and both completed successfully using latest main branch -- commit 9e1afd2fb36eeb4fdcd0f209735229cbf7df6802.

1. Could it be a networking fluke?
2. Did you try to re-run yarn install a second time?
3. According to Yarn's Documentation maybe your cached version of a package is bad for some reason? You could try running YARN_CHECKSUM_BEHAVIOR=reset yarn install to remove the potentially defective packages from your cache and re-fetch them.

[Montoya]

Oh I like that idea. I added checksumBehavior: update to .yarnrc.yml and that fixed the issue on the next yarn install but I will try pulling down the repo again and see if YARN_CHECKSUM_BEHAVIOR=reset yarn install also fixes it.

[Montoya]

Update: YARN_CHECKSUM_BEHAVIOR=reset yarn install does not fix the issue. YARN_CHECKSUM_BEHAVIOR=update yarn install does.

[Montoya]

Here's the result when running YARN_CHECKSUM_BEHAVIOR=update yarn install

[ziad-saab]

@Montoya does using update change any files in your working directory?

[Montoya]

Great question! Here are the changes in yarn.lock:

@ -5468,7 +5468,7 @@ __metadata:
"bignumber.js@git+https://github.com/frozeman/bignumber.js-nolookahead.git":
  version: 2.0.7
  resolution: "bignumber.js@https://github.com/frozeman/bignumber.js-nolookahead.git#commit=57692b3ecfc98bbdd6b3a516cb2353652ea49934"
- checksum: 8f9d21a8c7ef04b70098c49760679ee469e84f5469595c08aa01af47e1bae7703ae474f02abc89aa106e034e3b3faca0c0fd4faae72a037f2d45f4b26cbd6c3d
+ checksum: 514894e39b42ec471e07e86259ddae89ecb44333da5de1fcd8cdc9c14f97363bafec2209a6586c60cc2952f0678c9a99022224268e0510b17f0235e8b501294e
  languageName: node
  linkType: hard

@ -8522,7 +8522,7 @@ __metadata:
  dependencies:
    bn.js: ^4.11.8
    ethereumjs-util: ^6.0.0
- checksum: ae074be0bb012857ab5d3ae644d1163b908a48dd724b7d2567cfde309dc72222d460438f2411936a70dc949dc604ce1ef7118f7273bd525815579143c907e336
+ checksum: 03127d09960e5f8a44167463faf25b2894db2f746376dbb8195b789ed11762f93db9c574eaa7c498c400063508e9dfc1c80de2edf5f0e1406b25c87d860ff2f1
  languageName: node
  linkType: hard

[ziad-saab]

@Montoya that makes sense and is what I expected. I don't think using update is a valid fix though, because it means there's a discrepancy between the packages you get and the ones you're supposed to get.

Have you tried running yarn cache clean --all from the root of the project's working directory, then running yarn install again?

[Montoya]

OK I tried running yarn cache clean --all and then yarn install, and it still doesn't work.

(base) consensys-cm@Christians-MBP template-snap-monorepo-main % yarn cache clean --all
➤ YN0000: Done in 0s 437ms
(base) consensys-cm@Christians-MBP template-snap-monorepo-main % yarn install
➤ YN0000: ┌ Resolution step
➤ YN0002: │ eslint-config-react-app@npm:7.0.1 [3d5e3] doesn't provide @babel/plugin-syntax-flow (pace73), requested by eslint-plugin-flowtype
➤ YN0002: │ eslint-config-react-app@npm:7.0.1 [3d5e3] doesn't provide @babel/plugin-transform-react-jsx (pcdf2c), requested by eslint-plugin-flowtype
➤ YN0002: │ eth-block-tracker@npm:4.4.3 doesn't provide @babel/core (p2eb50), requested by @babel/plugin-transform-runtime
➤ YN0002: │ react-dev-utils@npm:12.0.1 doesn't provide typescript (p1c171), requested by fork-ts-checker-webpack-plugin
➤ YN0002: │ react-dev-utils@npm:12.0.1 doesn't provide webpack (p6531d), requested by fork-ts-checker-webpack-plugin
➤ YN0000: │ Some peer dependencies are incorrectly met; run yarn explain peer-requirements <hash> for details, where <hash> is the six-letter p-prefixed code
➤ YN0000: └ Completed
➤ YN0000: ┌ Fetch step
➤ YN0013: │ jest-environment-jsdom@npm:27.5.1 can't be found in the cache and w
➤ YN0013: │ jest-environment-node@npm:27.5.1 can't be found in the cache and wi
➤ YN0013: │ jest-get-type@npm:27.5.1 can't be found in the cache and will be fe
➤ YN0013: │ jest-get-type@npm:28.0.2 can't be found in the cache and will be fe
➤ YN0013: │ jest-haste-map@npm:27.5.1 can't be found in the cache and will be f
➤ YN0018: │ bignumber.js@https://github.com/frozeman/bignumber.js-nolookahead.git#commit=57692b3ecfc98bbdd6b3a516cb2353652ea49934: The remote archive doesn't match the expected checksum
➤ YN0013: │ json-stringify-safe@npm:5.0.1 can't be found in the cache and will 
➤ YN0013: │ json5@npm:1.0.1 can't be found in the cache and will be fetched fro
➤ YN0013: │ json5@npm:2.2.1 can't be found in the cache and will be fetched fro
➤ YN0013: │ jsonfile@npm:6.1.0 can't be found in the cache and will be fetched 
➤ YN0013: │ jsonify@npm:0.0.0 can't be found in the cache and will be fetched f
➤ YN0018: │ ethereumjs-abi@https://github.com/ethereumjs/ethereumjs-abi.git#commit=ee3994657fa7a427238e6ba92a84d0b529bbcde0: The remote archive doesn't match the expected checksum
➤ YN0013: │ yargs-parser@npm:21.0.1 can't be found in the cache and will be fet
➤ YN0013: │ yargs@npm:16.2.0 can't be found in the cache and will be fetched fr
➤ YN0013: │ yargs@npm:17.4.1 can't be found in the cache and will be fetched fr
➤ YN0013: │ yargs@npm:17.5.1 can't be found in the cache and will be fetched fr
➤ YN0013: │ yocto-queue@npm:0.1.0 can't be found in the cache and will be fetch
➤ YN0000: └ Completed in 34s 921ms
➤ YN0000: Failed with errors in 35s 65ms
(base) consensys-cm@Christians-MBP template-snap-monorepo-main % 

[ziad-saab]

It looks like the issue is happening with two packages that are being downloaded from GitHub instead of NPM. Can you look at the comment here (yarnpkg/berry#1142 (comment)) and see if that could fix your issue?

To be clearer, it looks like something is making your computer compute the checksums differently for these two packages. Unless you have a proxy to github.com, then the simplest explanation is you're downloading the same packages as me, but their checksum is being calculated differently. The comment I linked to might be one reason why.

[Montoya]

This is something for @GuillaumeRx to look at!

[ziad-saab]

@Montoya I believe it could be a setting on your machine, like the comment describes. Do you have a user-level gitattributes file on your machine?

[Montoya]

I am not aware of a gitattributes file on my machine. If I have one, I cannot find it.

[ziad-saab]

What's the output of running git config --global core.attributesFile in your terminal?

By the way if you're interested we can pair on this over Zoom. I'm pretty sure the issue is something on your machine.

[Montoya]

When I run that command I get no output or errors.

[ziad-saab]

There seems to be a much more extensive discussion here (yarnpkg/berry#2399), I'm currently looking into it both for myself and for this issue. It's very educational 🤓 I'll report what I find out.

[ziad-saab]

Turns out it's not so much "your computer" as "we have different computers / OSs". Explanations here:

• yarn pack produces slightly different .tgz files on different operating systems yarnpkg/berry#2774
• [Bug] Checksum mismatch for package referenced as "git+https://github.com..." yarnpkg/berry#2399

The gist: dependencies from GitHub get packed differently on different OSs, causing different checksums. This is bad, because:

• using update behavior will cause yarn.lock to be constantly re-committed, frustrating
• using ignore defeats the purpose of having these checksums in the first place

For now, to avoid constantly recommitting the lockfile, ignore is the best possible solution. We should monitor the two issues on Yarn's repo, and document this flaw in our README.

[Montoya]

I had a feeling it was this. Is it possible to just use ignore for those two files? At least to isolate the solution. Or does this mean checksums will be ignored for all the packages?

[ziad-saab]

I was planning to look into this exact question today. I'll report my findings

[ziad-saab]

From my research it looks like it's not possible to simply ignore the checksum for one or two packages. It's either all or nothing.

An alternative would be to commit the .yarn/cache to Git. But this has two issues that come with it:

1. The .yarn/cache directory is not small, and would have to be downloaded every time the repo is cloned
2. This repo is a template that will be used by other developers, which means we'd be "advocating" for this way of doing things. If a project opts out of committing the Yarn cache, they might run into the same issue.

Right now I can't seem to find a 100% foolproof way to do this. Wonder if @GuillaumeRx has any input. I'll keep looking.

[GuillaumeRx]

Found out that this is coming from web3 in @metamask/controllers . Yarn 3 has a strange behaviour, will be fixed directly in controllers repo since it's also affecting other repos.

[Montoya]

Looks like this PR fixes the issue: MetaMask/snaps#955

[FrederikBolding]

Looks like this PR fixes the issue: MetaMask/snaps-monorepo#955

No no, this has been fixed for a while. We just didn't realize.

[Montoya]

If you say so @FrederikBolding. I was still seeing the issue last week...