Cargo build without cargoSha256, use Cargo.lock instead

[nmattia]

Issue description

I’ve been experimenting with a new build platform that doesn’t require specifying a cargoSha256 but instead relies on the information contained in the Cargo.lock. There are some open issues, maybe someone here will have good solutions!

(note: happy to move this to discourse if needed)

The problems with the current rust platform (in particular with cargoSha256) is that it’s

1. unergonomic: every single dependency update requires the developer to update the cargoSha256. This means tweaking the existing checksum, re-building, copying the new hash, etc. This becomes a huge mess when lots of developers are working on the same cargo project.
2. not repeatable: if a developer forgets to update the hash, nix will use the old dependencies (nix store lookup by sha256) and in some cases the build may succeed nonetheless (think: security patch in a dependency, which doesn’t tinker with the crate’s interface). However a build on a different machine — where the vendored crates haven’t been cached yet — will result in a hash mismatch.

How

The idea is to leverage the “recently” added builtins.fromTOML to read the Cargo.lock in Nix directly and fetch all crates before the build even begins (I’ve done something similar for npm). The lockfile looks something like this:

[[package]]
name = "aho-corasick"
version = "0.6.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
 "memchr 2.2.0 (registry+https://github.com/rust-lang/crates.io-index)",
]
...

[metadata]
"checksum aho-corasick 0.6.10 (registry+https://github.com/rust-lang/crates.io-index)" = "81ce3d38065e618af2d7b77e10c5ad9a069859b4be3c2250f674af3840d9c8a5"

Quoting naersk (the POC for this new platform):

    # XXX: the actual crate format is not documented but in practice is a
    # gzipped tar; we simply unpack it and introduce a ".cargo-checksum.json"
    # file that cargo itself uses to double check the sha256

The corollary is that the lockfile contains all the information Nix needs in order to download dependencies (at least for crates coming from crates.io, see below for git dependencies).

Current status

The POC works surprisingly well. Most libs build out of the box but there are some issues that worry me a bit.

Libs that build

• https://github.com/rust-lang/rustfmt
• https://github.com/fastly/lucet
• https://github.com/phiresky/ripgrep-all
• https://github.com/rust-lang/rustlings
• https://github.com/pingcap/talent-plan
• https://github.com/target/

(I basically went through the list of trending rust repos)

Missing

• crate-level incrementality: currently all dependencies are built alongside the main crate. This is not too bad (rustc is pretty fast) but it could be improved.
• cargo doesn’t register checksums for git dependencies: this is a pretty nasty issue, because fetchFromGitHub and friends require a sha256. There are some workarounds:
  • use builtins.fetchgit ... which doesn’t require a sha256. The major drawbacks is that fetchgit will redownload the repo every hour and doesn’t support submodules. Moreover it’s somewhat ugly.
  • use cargoSha256 for git deps ... which is somewhat work ergonomic than having a single cargoSha256 for all deps, but still not ideal
  • track git dependencies outside of cargo ... e.g. “vendor” the dependencies through nix. The major drawback is that — while it’s fairly easy to do source replacement in a nix-build/derivation — it’s a pain to make sure that cargo (in a dev’s terminal) uses the same dependencies. There are some workarounds that can be implemented in a nix-shell (shellHook setting $CARGO_HOME with custom config, or shellHook that symlinks the store paths in $PWD) but it’s still very, very suboptimal.

CC @grahamc @basvandijk

[Hoverbear]

Oh my gosh this would be so handy to have official!

[stale]

I marked this as stale due to inactivity. → More info

[Hoverbear]

This is still important to me!

[Licenser]

I'd love to see this too 👍

[maddiemort]

I would also still really love to see this!

[nmattia]

Note: naersk was just moved to https://github.com/nix-community/naersk, so I guess we're one step closer!

[stale]

I marked this as stale due to inactivity. → More info

[Hoverbear]

This is still important to me!

[Stunkymonkey]

fyi: #221716