Don't check out the crates.io index locally #4026

alexcrichton · 2017-05-10T22:06:47Z

This commit moves working with the crates.io index to operating on the git
object layers rather than actually literally checking out the index. This is
aimed at two different goals:

Improving the on-disk file size of the registry
Improving cloning times for the registry as the index doesn't need to be
checked out

The on disk size of my registry folder of a fresh check out of the index went
form 124M to 48M, saving a good chunk of space! The entire operation took about
0.6s less on a Unix machine (out of 4.7s total for current Cargo). On Windows,
however, the clone operation went from 11s to 6.7s, a much larger improvement!

Closes #4015

rust-highfive · 2017-05-10T22:06:58Z

r? @brson

(rust_highfive has picked a reviewer for you, use r? to override)

alexcrichton · 2017-05-10T22:07:00Z

r? @matklad

Mark-Simulacrum · 2017-05-10T22:10:20Z

src/cargo/sources/registry/remote.rs

+
+    // Note that this `'static lifetime here is actually a lie, it's actually a
+    // borrow into the `repo` object below. We're guaranteed, though, that if
+    // filled in `tree` will be destroyed first, so this should be ok.


What guarantees that tree will be destroyed first? Unspecified drop order? In that case, maybe this is a good case for ManuallyDrop? Or is that not yet usable in cargo due to instability?

@alexcrichton ping. Should there be a crate for this sort of stuff? https://github.com/Kimundi/owning-ref-rs perhaps?

Oh sorry missed this! Yes the unspecified drop order is what guarantees this. Lots of projects are relying on this so I don't think it's necessary to go out of the way and use ManuallyDrop, and yeah I'd also prefer to keep Cargo on stable.

@matklad unfortunately that crate won't help as it's targeted at Rust pointers, whereas here it's all phantom lifetimes through libgit2 :(

Since it's in an Option, you could explicitly take() it first in an impl Drop for RemoteRegistry.

Indeed! I'll do that.

bors · 2017-05-10T23:26:04Z

☔ The latest upstream changes (presumably #4024) made this pull request unmergeable. Please resolve the merge conflicts.

matklad · 2017-05-11T12:07:00Z

src/cargo/sources/registry/index.rs

+                // interpretation of each line here and older cargo will simply
+                // ignore the new lines.
+                let lines = contents.split(|b| *b == b'\n')
+                                    .filter_map(|b| str::from_utf8(b).ok())


Ideally we should not swallow utf8-decoding errors here.

Or are we planing to switch to binary format some day?

Nah this was mostly inspired from discussion on the RFC about schema versioning. I have no plans to break this personally, but it seems reasonable to be somewhat defensive about future changes to the index just for maximal flexibility of future cargo's implementation.

I'm totally ok with self.parse_registry_package(line).ok(), it's only str::from_utf8(b).ok() that feels overly defensive.

Sounds reasonable!

matklad · 2017-05-11T12:09:12Z

src/cargo/sources/registry/local.rs

@@ -34,7 +35,11 @@ impl<'cfg> RegistryData for LocalRegistry<'cfg> {
        &self.index_path
    }

-    fn config(&self) -> CargoResult<Option<RegistryConfig>> {
+    fn load(&self, root: &Path, path: &str) -> CargoResult<Vec<u8>> {


Why not path: &Path? We convert str to Path both for Local and Remote registry anyway. Those slashes format!("{}/{}/{}", &fs_name[0..2], &fs_name[2..4], fs_name) make me nervous :)

And do we need root here? Can't we reconstruct it from index_path?

Ah my thinking of passing in root is that index_path returns a Filesystem which is an "unlocked path", but here we've always got a locked path (locked elsewhere) so looking at a Path is proof of that.

I was originally unsure what would happen if we take a \-separated path when we go down to libgit2, I'm not sure if it handles internally the slash differences. Only one way to find out!

matklad · 2017-05-11T12:18:26Z

src/cargo/sources/registry/remote.rs

+    // Note that this `'static lifetime here is actually a lie, it's actually a
+    // borrow into the `repo` object below. We're guaranteed, though, that if
+    // filled in `tree` will be destroyed first, so this should be ok.
+    tree: LazyCell<RefCell<Option<git2::Tree<'static>>>>,


Why do we need LazyCell on top of RefCell? LazyCell is useful to return &T and not Ref<T>, but we are returning a Ref anyway, so just RefCell<Option<git2::Tree<'static>>> should be enough levels of indirection...

Hm excellent point!

alexcrichton · 2017-05-11T14:51:22Z

Pushed some updates

matklad · 2017-05-11T14:58:27Z

src/cargo/sources/registry/remote.rs

+        let handle = ops::http_handle(self.config)?;
+        self.handle.fill(RefCell::new(handle)).ok().unwrap();
+        Ok(self.handle.borrow().unwrap())
+    }


Looks like easy and repo could use LazyCell::get_or_try_init instead of manually unwrapping things.

Aha yes indeed! I thought I tried that but thanks for catching

alexcrichton · 2017-05-11T19:08:02Z

Updated

matklad · 2017-05-11T19:10:53Z

LGTM, though there was seemingly legitimate failure on appveyor on the previous build.

alexcrichton · 2017-05-11T20:53:15Z

Bah looks like libgit2 cares about \ vs /

alexcrichton · 2017-05-11T20:53:29Z

@bors: r=matklad

bors · 2017-05-11T20:53:30Z

📌 Commit b7414ce has been approved by matklad

bors · 2017-05-11T22:04:17Z

🔒 Merge conflict

bors · 2017-05-11T22:05:32Z

☔ The latest upstream changes (presumably #4032) made this pull request unmergeable. Please resolve the merge conflicts.

This commit moves working with the crates.io index to operating on the git object layers rather than actually literally checking out the index. This is aimed at two different goals: * Improving the on-disk file size of the registry * Improving cloning times for the registry as the index doesn't need to be checked out The on disk size of my `registry` folder of a fresh check out of the index went form 124M to 48M, saving a good chunk of space! The entire operation took about 0.6s less on a Unix machine (out of 4.7s total for current Cargo). On Windows, however, the clone operation went from 11s to 6.7s, a much larger improvement! Closes rust-lang#4015

alexcrichton · 2017-05-11T22:05:53Z

@bors: r=matklad

bors · 2017-05-11T22:05:53Z

📌 Commit 15cc376 has been approved by matklad

bors · 2017-05-11T22:06:00Z

⌛ Testing commit 15cc376 with merge d8fa3eb...

Don't check out the crates.io index locally This commit moves working with the crates.io index to operating on the git object layers rather than actually literally checking out the index. This is aimed at two different goals: * Improving the on-disk file size of the registry * Improving cloning times for the registry as the index doesn't need to be checked out The on disk size of my `registry` folder of a fresh check out of the index went form 124M to 48M, saving a good chunk of space! The entire operation took about 0.6s less on a Unix machine (out of 4.7s total for current Cargo). On Windows, however, the clone operation went from 11s to 6.7s, a much larger improvement! Closes #4015

bors · 2017-05-12T00:37:56Z

☀️ Test successful - status-appveyor, status-travis
Approved by: matklad
Pushing d8fa3eb to master...

Allows for seamless ransition for when rust-lang/cargo#4026 lands Closes #32

rust-highfive assigned brson May 10, 2017

alexcrichton mentioned this pull request May 10, 2017

Use a bare clone of crates.io-index #4015

Closed

rust-highfive assigned matklad and unassigned brson May 10, 2017

Mark-Simulacrum reviewed May 10, 2017

View reviewed changes

alexcrichton force-pushed the bare-registry branch from 6bbd534 to dcbbfab Compare May 10, 2017 23:59

matklad reviewed May 11, 2017

View reviewed changes

alexcrichton force-pushed the bare-registry branch 2 times, most recently from 1ea815a to a4e850f Compare May 11, 2017 14:51

matklad reviewed May 11, 2017

View reviewed changes

alexcrichton force-pushed the bare-registry branch 2 times, most recently from e51d729 to c2e82c2 Compare May 11, 2017 19:07

alexcrichton force-pushed the bare-registry branch 2 times, most recently from 7ff4bca to b7414ce Compare May 11, 2017 20:52

alexcrichton force-pushed the bare-registry branch from b7414ce to 15cc376 Compare May 11, 2017 22:05

bors merged commit 15cc376 into rust-lang:master May 12, 2017

bors mentioned this pull request May 12, 2017

Support vendoring git repositories #3992

Merged

alexcrichton added the relnotes Release-note worthy label May 12, 2017

alexcrichton deleted the bare-registry branch May 12, 2017 16:49

ishitatsuyuki mentioned this pull request May 15, 2017

Incompatible with upcoming bare Cargo registry nabijaczleweli/cargo-update#32

Closed

nabijaczleweli added a commit to nabijaczleweli/cargo-update that referenced this pull request May 16, 2017

Read registry as a repository rather than a file tree

1ec0f2f

Allows for seamless ransition for when rust-lang/cargo#4026 lands Closes #32

Eh2406 mentioned this pull request Oct 12, 2017

Don't need to check out the repo to find crates frewsxcv/rust-crates-index#17

Closed

est31 mentioned this pull request Jul 2, 2018

New RFC: signing registry index commits rust-lang/rfcs#2474

Closed

ehuss added this to the 1.19.0 milestone Feb 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Don't check out the crates.io index locally #4026

Don't check out the crates.io index locally #4026

alexcrichton commented May 10, 2017

rust-highfive commented May 10, 2017

alexcrichton commented May 10, 2017

Mark-Simulacrum May 10, 2017

matklad May 11, 2017

alexcrichton May 11, 2017

cuviper May 11, 2017

alexcrichton May 11, 2017

bors commented May 10, 2017

matklad May 11, 2017

matklad May 11, 2017

alexcrichton May 11, 2017

matklad May 11, 2017

alexcrichton May 11, 2017

matklad May 11, 2017

matklad May 11, 2017

alexcrichton May 11, 2017

matklad May 11, 2017 •

edited

Loading

alexcrichton May 11, 2017

alexcrichton commented May 11, 2017

matklad May 11, 2017

alexcrichton May 11, 2017

alexcrichton commented May 11, 2017

matklad commented May 11, 2017

alexcrichton commented May 11, 2017

alexcrichton commented May 11, 2017

bors commented May 11, 2017

bors commented May 11, 2017

bors commented May 11, 2017

alexcrichton commented May 11, 2017

bors commented May 11, 2017

bors commented May 11, 2017

bors commented May 12, 2017

Don't check out the crates.io index locally #4026

Don't check out the crates.io index locally #4026

Conversation

alexcrichton commented May 10, 2017

rust-highfive commented May 10, 2017

alexcrichton commented May 10, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bors commented May 10, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matklad May 11, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexcrichton commented May 11, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexcrichton commented May 11, 2017

matklad commented May 11, 2017

alexcrichton commented May 11, 2017

alexcrichton commented May 11, 2017

bors commented May 11, 2017

bors commented May 11, 2017

bors commented May 11, 2017

alexcrichton commented May 11, 2017

bors commented May 11, 2017

bors commented May 11, 2017

bors commented May 12, 2017

matklad May 11, 2017 •

edited

Loading