Issues regarding lockfile hashes and the recommended zero-install workflow

[purepani]

Recently, I've been investigating trying to get yarn(and nodejs) projects into nixpkgs more effectively. To get to the point, it would be useful to have the integrity hashes in the yarn lockfiles to do that easily. This was the case in the v1 spec of the lock file and is no longer the case. Now the only hash that is used is one that is generated based on the zip file on a local computer; since zip files are inherently non-deterministic(See #6068 and #6105), this ends up being useless for the purposes of reproducibility.

On https://yarnpkg.com/cli/install, it is mentioned that yarn install is not part of the recommended work flow, and that .pnp.cjs and the cache should be commited. Unfortunately, the combination of these 2 things, plus the fact that a lot of repos are transitioning from to yarn berry seems to mean that the zero-install workflow isn't used super often; this means that repos that previously were fully reproducible with just the yarn.lock file are no longer reproducible.

That confusion seems to be causing other issues as well(see #6105). If you aren't using zero-install and commiting the files, there's no reproducibility using yarn.

Proposed solutions:

• Require zero install. I don't think this is a good idea, since it would be completely breaking.
• Add integrity hashes back into the lock file. This would be significantly easier, and would at the very least make the repo reproducible.
• Generate a normal package-lock.json. This is probably stupid but it's technically a solution.

The only issue I can think of with option 2 is that you probably can't have both the integrity hash and the checksum in the file because you can't guarantee that they would refer to the same thing. The only way around that as far as I can tell, is to have an option for enabling zero-installs, and storing the checksum or the integrity hash depending on the setting. If the cache and .pnp.cjs files are not in the git tree, then warnings should probably be popping up as well.

[arcanis]

seems to mean that the zero-install workflow isn't used super often; this means that repos that previously were fully reproducible with just the yarn.lock file are no longer reproducible

If you aren't using zero-install and commiting the files, there's no reproducibility using yarn.

What makes you think any of that?

Add integrity hashes back into the lock file.

They're already there (and never left). It doesn't have anything to do with reproducibility though.

[purepani]

Previously, the lock files for yarn(before yarn berry) included the integrity hash from npm. Now they do not. It is now impossible to download a package from npm, calculate a sha-512 hash, and compare it to the sha-512 hash from a yarn lock file. This was previously possible. The 2 issues I linked are effects of that. One, where differences in the compression level calculates a different hash, and another where being on a different OS calculates a different hash.

I wa using the terms a bit loosely: integrity hash was referring to the hash that is in npm, and the hash that was previously stored in the yarn lock file.

Whereas right now, the hash(which I was referring to as checksum to match the field name) seems to be calculated after rezipping the package.

Do you agree that it is currently not possible to guarantee you can calculate the same hash for the same package using yarn on different systems? Because that's the main issue (aside from also not being the same as the one you can see from the npm registry).

If that's something you don't agree with, see the 2 linked issues. Otherwise, is there any reason there can't be a fully reproducable hash like before?

[arcanis]

Do you agree that it is currently not possible to calculate the same hash for the same package using yarn on different systems?

No, it's possible. We include our own copy of zlib + libzip that generates the same output regardless of the machine / Node version (unlike the native zlib module, whose output may randomly change).

The issue in #6105 is caused when using the file: protocol (or sometimes git: protocol) on systems that assign different metadata to the packed files. That's not directly related to Yarn (although we're always open to ways to solve that), and isn't something that using the hash from npm would solve (since file: and git: dependencies don't go through npm).

[purepani]

Turns out I can't read, and also that the compression level is actually stored in the lock file
sorry about that