Detect truncated DepGraph files #122245

saethlin · 2024-03-09T17:55:17Z

I suspect that the following issues are caused by truncated incr comp files:

We fail with an allocation failure or capacity overflow in this case because we assume that the ending bytes of an DepGraph file are the lengths of arrays. If the file has somehow been truncated then the ending bytes are probably some of our varint encoding, which tries to eliminate zero bytes, so interpreting a random 8 bytes as an array length has a very high chance of producing a byte capacity over isize::MAX.

Now theoretically since #119510 merged I have fixed the out-of-disk issues and yet in #120894 (comment) I still see some decoding failures that look like out-of-disk ICEs, for example https://crater-reports.s3.amazonaws.com/beta-1.77-1/beta-2024-02-10/gh/scottfones.aoc_2022/log.txt

So this PR should ensure that we get an ICE that clearly identifies if the file in question is truncated.

rustbot · 2024-03-09T17:55:24Z

r? @lcnr

rustbot has assigned @lcnr.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

cjgillot · 2024-03-09T18:27:49Z

compiler/rustc_incremental/src/persist/file_format.rs

+        let mut len = [0u8; 8];
+        file.read_exact(&mut len)?;
+        let len_in_header = u64::from_le_bytes(len);
+        assert_eq!(len_in_header, fs::metadata(path)?.len());


Should this be an ICE or a fatal error? Do we suggest a cargo clean?

My personal suggestion (granted, as a non-contributor, so worth a grain of salt) would be to suggest a cargo clean with an ICE if possible. cargo clean does appear to unstick and allow later compilation to continue normally, and a fatal error doesn't seem appropriate/the right severity.

If the assertion I'm trying to add in this PR ever fails, we have a new defect somewhere. We should ICE.

If we want to encourage people to paper over the problem with cargo clean we should just remove the file for them.

Noratrieb · 2024-03-09T19:14:34Z

compiler/rustc_incremental/src/persist/file_format.rs

@@ -35,6 +36,8 @@ pub(crate) fn write_file_header(stream: &mut FileEncoder, sess: &Session) {
    assert_eq!(rustc_version.len(), (rustc_version.len() as u8) as usize);
    stream.emit_raw_bytes(&[rustc_version.len() as u8]);
    stream.emit_raw_bytes(rustc_version.as_bytes());
+    // Reserve space for the u64 where we will later write in the expected total size of this file.
+    stream.emit_raw_bytes(&0u64.to_le_bytes());


Would it be simpler to just append some magic bytes like b"the-end-of-rust" or something like that? Then all the metadata calls wouldn't be needed.

I'd rather report the expected and actual numbers so that we can tell if the dep graph is truncated, unexpectedly large, and about how big it is. Some systems have file size limits that it's plausible we run into, and having a number would let us know about that.

saethlin · 2024-03-09T19:42:47Z

Turns out this is the hard part:

rust/compiler/rustc_incremental/src/persist/save.rs

Lines 169 to 181 in 8401645

    
           file_format::write_file_header(&mut encoder, sess); 
        
           // First encode the commandline arguments hash 
        
           sess.opts.dep_tracking_hash(false).encode(&mut encoder); 
        
           Some(DepGraph::new( 
        
               &sess.prof, 
        
               prev_graph, 
        
               prev_work_products, 
        
               encoder, 
        
               sess.opts.unstable_opts.query_dep_graph, 
        
               sess.opts.unstable_opts.incremental_info, 
        
           ))

the DepGraph that's encoded here doesn't go through save_in, which is truly awful.

lcnr · 2024-03-11T09:07:00Z

r? @cjgillot feel free to reassign

lcnr · 2024-03-11T14:22:51Z

r? compiler

petrochenkov · 2024-03-12T13:57:39Z

I think it's fine to land this as an ICE first and then think what to do next when some actual examples of that ICE are found in practice.
@bors r+

bors · 2024-03-12T13:57:43Z

📌 Commit 11f8866 has been approved by petrochenkov

It is now in the queue for this repository.

…kingjubilee Rollup of 12 pull requests Successful merges: - rust-lang#121754 ([bootstrap] Move the `split-debuginfo` setting to the per-target section) - rust-lang#121953 (Add tests for the generated assembly of mask related simd instructions.) - rust-lang#122081 (validate `builder::PATH_REMAP`) - rust-lang#122245 (Detect truncated DepGraph files) - rust-lang#122354 (bootstrap: Don't eagerly format verbose messages) - rust-lang#122355 (rustdoc: fix up old test) - rust-lang#122363 (tests: Add ui/attributes/unix_sigpipe/unix_sigpipe-str-list.rs) - rust-lang#122366 (Fix stack overflow with recursive associated types) - rust-lang#122377 (Fix discriminant_kind copy paste from the pointee trait case) - rust-lang#122378 (Properly rebuild rustbooks) - rust-lang#122380 (Fix typo in lib.rs of proc_macro) - rust-lang#122381 (llvm-wrapper: adapt for LLVM API changes) r? `@ghost` `@rustbot` modify labels: rollup

Rollup merge of rust-lang#122245 - saethlin:check-dep-graph-size, r=petrochenkov Detect truncated DepGraph files I suspect that the following issues are caused by truncated incr comp files: * rust-lang#120582 * rust-lang#121499 * rust-lang#122210 We fail with an allocation failure or capacity overflow in this case because we assume that the ending bytes of an DepGraph file are the lengths of arrays. If the file has somehow been truncated then the ending bytes are probably some of our varint encoding, which tries to eliminate zero bytes, so interpreting a random 8 bytes as an array length has a very high chance of producing a byte capacity over `isize::MAX`. Now theoretically since rust-lang#119510 merged I have fixed the out-of-disk issues and yet in rust-lang#120894 (comment) I still see some decoding failures that look like out-of-disk ICEs, for example https://crater-reports.s3.amazonaws.com/beta-1.77-1/beta-2024-02-10/gh/scottfones.aoc_2022/log.txt So this PR should ensure that we get an ICE that clearly identifies if the file in question is truncated.

rustbot assigned lcnr Mar 9, 2024

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Mar 9, 2024

This comment has been minimized.

Sign in to view

saethlin force-pushed the check-dep-graph-size branch from a1ad6ca to c0b1ccf Compare March 9, 2024 18:08

This comment has been minimized.

Sign in to view

cjgillot reviewed Mar 9, 2024

View reviewed changes

Noratrieb reviewed Mar 9, 2024

View reviewed changes

saethlin marked this pull request as draft March 9, 2024 20:16

saethlin added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Mar 9, 2024

Detect truncated incr comp files

11f8866

saethlin force-pushed the check-dep-graph-size branch from c0b1ccf to 11f8866 Compare March 9, 2024 20:44

rustbot added the A-query-system Area: The rustc query system (https://rustc-dev-guide.rust-lang.org/query.html) label Mar 9, 2024

saethlin marked this pull request as ready for review March 9, 2024 20:56

saethlin added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Mar 10, 2024

saethlin changed the title ~~Detect truncated incr comp files~~ Detect truncated DepGraph files Mar 10, 2024

rustbot assigned cjgillot and unassigned lcnr Mar 11, 2024

rustbot assigned petrochenkov and unassigned cjgillot Mar 11, 2024

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Mar 12, 2024

workingjubilee mentioned this pull request Mar 12, 2024

Rollup of 12 pull requests #122389

Merged

bors merged commit f54350a into rust-lang:master Mar 12, 2024
11 checks passed

rustbot added this to the 1.78.0 milestone Mar 12, 2024

saethlin deleted the check-dep-graph-size branch March 12, 2024 19:03

saethlin mentioned this pull request Mar 26, 2024

using mold results into panic #123073

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Detect truncated DepGraph files #122245

Detect truncated DepGraph files #122245

saethlin commented Mar 9, 2024 •

edited

Loading

rustbot commented Mar 9, 2024

This comment has been minimized.

This comment has been minimized.

cjgillot Mar 9, 2024

amyipdev Mar 9, 2024

saethlin Mar 9, 2024

Noratrieb Mar 9, 2024 •

edited

Loading

saethlin Mar 9, 2024

saethlin commented Mar 9, 2024

lcnr commented Mar 11, 2024

lcnr commented Mar 11, 2024

petrochenkov commented Mar 12, 2024

bors commented Mar 12, 2024

Detect truncated DepGraph files #122245

Detect truncated DepGraph files #122245

Conversation

saethlin commented Mar 9, 2024 • edited Loading

rustbot commented Mar 9, 2024

This comment has been minimized.

This comment has been minimized.

cjgillot Mar 9, 2024

Choose a reason for hiding this comment

amyipdev Mar 9, 2024

Choose a reason for hiding this comment

saethlin Mar 9, 2024

Choose a reason for hiding this comment

Noratrieb Mar 9, 2024 • edited Loading

Choose a reason for hiding this comment

saethlin Mar 9, 2024

Choose a reason for hiding this comment

saethlin commented Mar 9, 2024

lcnr commented Mar 11, 2024

lcnr commented Mar 11, 2024

petrochenkov commented Mar 12, 2024

bors commented Mar 12, 2024

saethlin commented Mar 9, 2024 •

edited

Loading

Noratrieb Mar 9, 2024 •

edited

Loading