tree-wide: convert rust with git deps to importCargoLock #221716

yu-re-ka · 2023-03-17T19:23:38Z

Description of changes

Less invasive alternative to #217084

Things done

winterqt · 2023-03-17T20:07:11Z

0319ecc should explain why, at least in the commit message, as it does technically work. I assume you just want to discourage its use?

yu-re-ka · 2023-03-17T21:16:32Z

Well, with cargo 1.68 and above we can not rely on it being reproducible with git dependencies, as there is extended logic involved now which might change in future versions, so we would have to regenerate the FOD hashes on every update.

The assumption is that cargo vendor is still reproducible when no git dependencies are involved.

winterqt · 2023-03-17T21:27:57Z

Please re-read my comment -- that's exactly what I'm trying to say; it technically works, but we don't want anyone to actually use it. This should be made more clear, at least in the commit message, IMO.

zowoq · 2023-03-18T01:46:21Z

There are several lockfiles committed without a corresponding nix package change?

e.g:

squeekboard/Cargo.lock
gnome-podcats/Cargo.lock
emblem/Cargo.lock
rnote/Cargo.lock
citations/Cargo.lock
newsflash/Cargo.lock
warp/Cargo.lock

oxalica · 2023-03-18T10:39:54Z

Nixpkgs size increase of this PR is ~5% before compression, ~6% after compression.

$ fd '^Cargo.lock$' -X cat | wc -c
8.8Mi

$ git archive --format=tar HEAD~3 | wc -c
162600960
$ git archive --format=tar HEAD | wc -c
170997760

$ git archive --format=tar.gz HEAD~3 | wc -c
34881234
$ git archive --format=tar.gz HEAD | wc -c
37023424

yu-re-ka · 2023-03-18T15:33:19Z

There are several lockfiles committed without a corresponding nix package change?

good find, these should be fixed now

zowoq · 2023-03-26T00:40:45Z

pkgs/build-support/rust/fetch-cargo-tarball/default.nix

@@ -76,6 +76,17 @@ in stdenv.mkDerivation ({
    # Override the `http.cainfo` option usually specified in `.cargo/config`.
    export CARGO_HTTP_CAINFO=${cacert}/etc/ssl/certs/ca-bundle.crt

+    if grep --pcre2 '^source = "(?!registry)' Cargo.lock; then


grep: unrecognized option '--pcre2'

perhaps we can just do ^source = "git\+ since we don't really know how to deal with non-git deps anyway?

That was my alias grep=rg in my environment 😅

The reason is that we can not expect the extended logic run on git dependencies starting from Cargo 1.68 to be reproducible in future versions, and thus the output hash would not be sufficiently stable. rust-lang/cargo#11414

superherointj · 2023-03-27T22:32:51Z

Noticed a spike averaging ~6.5 seconds in eval times at nixpkgs. On bisecting found:

Preceding commit to this PR:

2f0ff93 Merge pull request #222585 from fabaff/yara-bump
- real 1m8,129s
  user 1m5,506s
  sys 0m2,620s

At this PR:

f8cbc3c tree-wide: convert rust with git deps to importCargoLock
- real 1m14,646s
  user 1m12,097s
  sys 0m2,546s

Command used for testing:

time nix-env --query --available --out-path --file ./. > /dev/null

Mic92 · 2023-03-29T06:22:47Z

Also around 10GB of RAM on x86_64-linux. Haven't compared what we had before.

alyssais · 2023-03-29T09:01:23Z

That's really weird, considering that in #217084 we found that converting all Rust packages rather than just ones with git deps had almost no impact on CPU time, and a similarly low impact on memory. cc @winterqt

alyssais · 2023-03-29T09:49:28Z

OfBorg also found a much bigger evaluation performance difference than we were expecting from Winter's research: https://github.com/NixOS/nixpkgs/runs/12279178729

kirillrdy · 2023-12-28T00:36:55Z

@yu-re-ka apologies if this has been answered somewhere, but is there a reason for not using Cargo.lock from ${src}/Cargo.lock ( where possible of course )

kirillrdy · 2023-12-28T01:22:58Z

@yu-re-ka apologies if this has been answered somewhere, but is there a reason for not using Cargo.lock from ${src}/Cargo.lock ( where possible of course )

I think I've found answer,

during evaluation because the option 'allow-import-from-derivation' is disabled

yu-re-ka requested review from piegamesde, mmahut, RaghavSood, figsoda, jonringer, teto, zowoq, winterqt and FRidh as code owners March 17, 2023 19:23

github-actions bot added 6.topic: python 6.topic: rust 6.topic: vim labels Mar 17, 2023

ofborg bot added 10.rebuild-darwin: 11-100 10.rebuild-linux: 101-500 labels Mar 17, 2023

yu-re-ka force-pushed the git-deps-import-cargo-lock branch 2 times, most recently from 33f4337 to cbd5504 Compare March 18, 2023 14:41

yu-re-ka changed the base branch from staging to master March 18, 2023 15:37

yu-re-ka force-pushed the git-deps-import-cargo-lock branch 3 times, most recently from 00a240f to c027ea2 Compare March 18, 2023 16:11

ofborg bot added 10.rebuild-darwin: 101-500 and removed 10.rebuild-darwin: 11-100 labels Mar 18, 2023

yu-re-ka force-pushed the git-deps-import-cargo-lock branch from d01de82 to 072690c Compare March 18, 2023 18:45

ofborg bot added the 2.status: merge conflict label Mar 18, 2023

yu-re-ka force-pushed the git-deps-import-cargo-lock branch from ab19193 to 173dced Compare March 26, 2023 00:30

zowoq reviewed Mar 26, 2023

View reviewed changes

yu-re-ka force-pushed the git-deps-import-cargo-lock branch from 173dced to c08cd7e Compare March 26, 2023 00:46

yu-re-ka added 3 commits March 26, 2023 01:52

rustPlatform.fetchCargoTarball: fail on git dependencies

4329de1

The reason is that we can not expect the extended logic run on git dependencies starting from Cargo 1.68 to be reproducible in future versions, and thus the output hash would not be sufficiently stable. rust-lang/cargo#11414

docs/rust: add note about git dependencies

f3a1640

tree-wide: convert rust with git deps to importCargoLock

f8cbc3c

yu-re-ka force-pushed the git-deps-import-cargo-lock branch from c08cd7e to f8cbc3c Compare March 26, 2023 00:52

zowoq merged commit d94076a into NixOS:master Mar 26, 2023

yu-re-ka deleted the git-deps-import-cargo-lock branch March 26, 2023 06:33

tomhoule mentioned this pull request Mar 29, 2023

prisma-engines: 4.11.0 -> 4.12.0 #223700

Merged

12 tasks

lilyinstarlight mentioned this pull request Mar 30, 2023

matrix-sdk-crypto-nodejs: 0.1.0-beta.2 -> 0.1.0-beta.3 #220317

Merged

12 tasks

justinas mentioned this pull request Apr 2, 2023

teleport: 12.1.0 -> 12.1.5 #224379

Merged

12 tasks

euank mentioned this pull request Apr 5, 2023

anki: 2.1.15 -> 2.1.60 #221229

Merged

11 tasks

huantianad mentioned this pull request Apr 27, 2023

activitywatch: init at 0.12.2 #202935

Merged

17 tasks

kira-bruneau mentioned this pull request May 4, 2023

newsflash: 2.2.4 -> 2.3.0 #229970

Merged

12 tasks

Stunkymonkey mentioned this pull request May 4, 2023

Cargo build without cargoSha256, use Cargo.lock instead #63653

Open

raphaelr mentioned this pull request Jun 6, 2023

treewide: convert remaining rust packages with git dependencies to importCargoLock #236308

Closed

12 tasks

winterqt mentioned this pull request Jul 31, 2023

rustc: 1.67.1 -> 1.68.1 #220373

Merged

12 tasks

pbsds mentioned this pull request Aug 7, 2023

pagefind: init at 1.0.3 #247773

Merged

12 tasks

kira-bruneau mentioned this pull request Aug 22, 2023

texlab: 5.9.0 -> 5.9.2 #250624

Merged

12 tasks

ahirner mentioned this pull request Dec 27, 2023

edgedb: 3.4.0 -> 4.0.2 #277193

Merged

13 tasks

amesgen mentioned this pull request Apr 26, 2024

matrix-synapse-tools.rust-synapse-compress-state: 0.1.3 -> 0.1.4 #303704

Merged

13 tasks

superherointj mentioned this pull request Jun 17, 2024

Nixpkgs eval time is increasing too fast #320528

Open

superherointj mentioned this pull request Jul 14, 2024

Cargo.lock considered harmful #327063

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tree-wide: convert rust with git deps to importCargoLock #221716

tree-wide: convert rust with git deps to importCargoLock #221716

yu-re-ka commented Mar 17, 2023 •

edited

Loading

winterqt commented Mar 17, 2023

yu-re-ka commented Mar 17, 2023

winterqt commented Mar 17, 2023

zowoq commented Mar 18, 2023

oxalica commented Mar 18, 2023 •

edited

Loading

yu-re-ka commented Mar 18, 2023

zowoq Mar 26, 2023

figsoda Mar 26, 2023 •

edited

Loading

yu-re-ka Mar 26, 2023

superherointj commented Mar 27, 2023 •

edited

Loading

Mic92 commented Mar 29, 2023

alyssais commented Mar 29, 2023 •

edited

Loading

alyssais commented Mar 29, 2023

kirillrdy commented Dec 28, 2023

kirillrdy commented Dec 28, 2023

tree-wide: convert rust with git deps to importCargoLock #221716

tree-wide: convert rust with git deps to importCargoLock #221716

Conversation

yu-re-ka commented Mar 17, 2023 • edited Loading

Description of changes

Things done

winterqt commented Mar 17, 2023

yu-re-ka commented Mar 17, 2023

winterqt commented Mar 17, 2023

zowoq commented Mar 18, 2023

oxalica commented Mar 18, 2023 • edited Loading

yu-re-ka commented Mar 18, 2023

zowoq Mar 26, 2023

Choose a reason for hiding this comment

figsoda Mar 26, 2023 • edited Loading

Choose a reason for hiding this comment

yu-re-ka Mar 26, 2023

Choose a reason for hiding this comment

superherointj commented Mar 27, 2023 • edited Loading

Mic92 commented Mar 29, 2023

alyssais commented Mar 29, 2023 • edited Loading

alyssais commented Mar 29, 2023

kirillrdy commented Dec 28, 2023

kirillrdy commented Dec 28, 2023

yu-re-ka commented Mar 17, 2023 •

edited

Loading

oxalica commented Mar 18, 2023 •

edited

Loading

figsoda Mar 26, 2023 •

edited

Loading

superherointj commented Mar 27, 2023 •

edited

Loading

alyssais commented Mar 29, 2023 •

edited

Loading