Allocate HIR on an arena 1/4 #66931

cjgillot · 2019-12-01T12:09:16Z

This PR is the first in a series of 4, aiming at allocating the HIR on an arena, as a memory optimisation.

This first PR lays the groundwork and migrates some low-hanging fruits.
The second PR will migrate hir::Expr, hir::Pat and related.
The third PR will migrate hir::Ty and related.
The final PR will be dedicated to eventual cleanups.

In order to make the transition as gradual as possible, some lowering routines receive Box-allocated data and move it into the arena. This is a bit wasteful, but hopefully temporary.
Nonetheless, special care should be taken to avoid double arena allocations.

Work mentored by @Zoxc.

matthewjasper · 2019-12-01T12:10:46Z

cc @rust-lang/compiler

Zoxc · 2019-12-01T12:25:26Z

r? @eddyb

Zoxc · 2019-12-01T12:27:47Z

src/librustc/arena.rs

@@ -123,6 +122,10 @@ macro_rules! arena_types {
            [few] crate_variances: rustc::ty::CrateVariancesMap<'tcx>,
            [few] inferred_outlives_crate: rustc::ty::CratePredicatesMap<'tcx>,
            [] upvars: rustc_data_structures::fx::FxIndexMap<rustc::hir::HirId, rustc::hir::Upvar>,
+            // HIR nodes arenas


Maybe call this HIR types and put a newline above it to visually separate it from the rest.

Zoxc · 2019-12-01T12:31:16Z

src/librustc/hir/intravisit.rs

    visitor.visit_mod(&krate.module, krate.span, CRATE_HIR_ID);
-    walk_list!(visitor, visit_attribute, &krate.attrs);
-    walk_list!(visitor, visit_macro_def, &krate.exported_macros);
+    walk_list!(visitor, visit_attribute, krate.attrs);


I'm curious, why didn't &krate.attrs still work?

I changed the HirVec<T> to be &'hir [T]. Using &krate.attrs used to make a &[T], and now makes a &&[T].

Zoxc · 2019-12-01T12:39:36Z

src/librustc/hir/lowering.rs

        /// Full-crate AST visitor that inserts into a fresh
        /// `LoweringContext` any information that may be
        /// needed from arbitrary locations in the crate,
        /// e.g., the number of lifetime generic parameters
        /// declared for every type and trait definition.
-        struct MiscCollector<'tcx, 'interner> {
-            lctx: &'tcx mut LoweringContext<'interner>,
+        struct MiscCollector<'tcx, 'interner, 'hir> {


The names of these lifetimes are quite misleading. I think they should be named MiscCollector<'misc, 'lowering, 'hir> instead.

src/librustc/hir/lowering.rs

Zoxc · 2019-12-01T12:54:34Z

src/librustc/hir/lowering/item.rs

@@ -25,11 +25,11 @@ use syntax_pos::Span;

 use rustc_error_codes::*;

-pub(super) struct ItemLowerer<'tcx, 'interner> {
-    pub(super) lctx: &'tcx mut LoweringContext<'interner>,
+pub(super) struct ItemLowerer<'tcx, 'interner, 'hir> {


This could be named ItemLowerer<'a, 'lowering, 'hir>.

Zoxc · 2019-12-01T12:56:52Z

src/librustc/hir/print.rs

@@ -107,7 +107,7 @@ pub const INDENT_UNIT: usize = 4;
 /// it can scan the input text for comments to copy forward.
 pub fn print_crate<'a>(cm: &'a SourceMap,
                       sess: &ParseSess,
-                       krate: &hir::Crate,
+                       krate: &hir::Crate<'a>,


This could probably be &hir::Crate<'_>?

Centril · 2019-12-01T13:03:15Z

src/librustc/hir/lowering/item.rs

-                            ImplTraitContext::Disallowed(ImplTraitPosition::Binding)
-                        }
-                    ),
+                    self.arena.alloc(ty.into_inner()),


This feels noisy and repetitive. Could we bake self.arena.alloc(ty.into_inner()) into lower_ty or if not, for whatever reason, add a lower_ty_arena?

I'd expect that to happen on the 3rd PR which moves Ty to the arena. I guess @cjgillot moved the fields of each type to the arena too so these show up as temporary changes.

There doesn't seem to be that many fields that's moved to &'hir Ty though. I'd just keep P<Ty> for this PR to avoid unnecessary changes.

Indeed, a lot of those disappear in #66942. I plan to do the final cleanup in PR 4/4.

After discussing with @pnkfelix, I remembered another reason to migrate eagerly the P<Ty> into &'hir Ty. If I do not do that, I get unused lifetime parameter errors in intermediate commits (ex: TraitItemKind). This keeps me from splitting in compiling commits.

Zoxc · 2019-12-01T13:56:54Z

src/librustc/arena.rs

@@ -125,7 +125,12 @@ macro_rules! arena_types {
            // HIR nodes arenas
            [few] hir_forest: rustc::hir::map::Forest<$tcx>,
            [] attribute: syntax::ast::Attribute,
+            [] global_asm: rustc::hir::GlobalAsm,


This could have a few attribute since they're likely to be rare.

Zoxc · 2019-12-01T13:58:09Z

src/librustc/arena.rs

+            // HIR nodes arenas
+            [few] hir_forest: rustc::hir::map::Forest<$tcx>,
+            [] attribute: syntax::ast::Attribute,
+            [] macro_def: rustc::hir::MacroDef,


This should probably have a few attribute too.

Zoxc · 2019-12-01T15:27:57Z

src/librustc/hir/lowering/item.rs

        let def_id = self.resolver.definitions().local_def_id(i.id);
        hir::ForeignItem {
            hir_id: self.lower_node_id(i.id),
            ident: i.ident,
-            attrs: self.lower_attrs(&i.attrs),
+            attrs: self.lower_attrs_extendable(&i.attrs),


This seems a bit odd. Shouldn't lower_attrs_extendable still return a Vec?

Zoxc · 2019-12-01T15:35:32Z

src/librustc/hir/lowering.rs

-            .iter()
-            .map(|a| self.lower_attr(a))
-            .collect()
+    fn lower_attrs_extendable(&mut self, attrs: &[Attribute]) -> &'hir [Attribute] {


This should still return Vec<Attribute> since callers may want to extend it.

lower_attrs is the function which should return &'hir [Attribute].

I suggest you add a lower_attrs_arena variant which returns &'hir [Attribute] which can be removed when all the attributes are migrated over.

Zoxc · 2019-12-01T16:15:38Z

src/librustc/hir/lowering/item.rs

@@ -1042,7 +1042,7 @@ impl LoweringContext<'_, 'hir> {

    fn lower_body(
        &mut self,
-        f: impl FnOnce(&mut LoweringContext<'_, '_>) -> (HirVec<hir::Param>, hir::Expr),
+        f: impl for<'tcx> FnOnce(&mut LoweringContext<'_, 'tcx>) -> (&'tcx [hir::Param], hir::Expr),


We don't want to use for<'tcx> here.

bors · 2019-12-02T06:18:44Z

☔ The latest upstream changes (presumably #66944) made this pull request unmergeable. Please resolve the merge conflicts.

cjgillot · 2019-12-03T07:20:47Z

Rebased with review comments.

eddyb

r=me if no concerns remain from @rust-lang/compiler (also, feel free to ping/PM me on Discord or Zulip in the future, when each PR is ready)

bors · 2019-12-09T01:59:31Z

☔ The latest upstream changes (presumably #66984) made this pull request unmergeable. Please resolve the merge conflicts.

nikomatsakis · 2019-12-09T21:00:30Z

My take is that we should land this. Maybe we can nominate for a quick 👍 at triage meeting? Seems like it would be good to get some kind of perf numbers, too. Let's kick that off.

@bors try

bors · 2019-12-09T21:00:45Z

⌛ Trying commit 0c4b8eee9c9e63f818861bec48e6fd356daf4166 with merge 963a1868bdc29bd60b42fc3c9705b6b186b61927...

cjgillot · 2019-12-21T22:43:18Z

Rebased.

Zoxc · 2019-12-22T07:37:06Z

@bors retry

Zoxc · 2019-12-22T07:37:44Z

@bors r=eddyb

bors · 2019-12-22T07:37:45Z

📌 Commit baa49b2 has been approved by eddyb

bors · 2019-12-22T07:37:46Z

🌲 The tree is currently closed for pull requests below priority 100, this pull request will be tested once the tree is reopened

matthewjasper · 2019-12-22T10:30:46Z

@bors p=100

bors · 2019-12-22T10:30:58Z

⌛ Testing commit baa49b2 with merge 26286c7...

@Zoxc

Allocate HIR on an arena 1/4 This PR is the first in a series of 4, aiming at allocating the HIR on an arena, as a memory optimisation. 1. This first PR lays the groundwork and migrates some low-hanging fruits. 2. The second PR will migrate `hir::Expr`, `hir::Pat` and related. 3. The third PR will migrate `hir::Ty` and related. 4. The final PR will be dedicated to eventual cleanups. In order to make the transition as gradual as possible, some lowering routines receive `Box`-allocated data and move it into the arena. This is a bit wasteful, but hopefully temporary. Nonetheless, special care should be taken to avoid double arena allocations. Work mentored by @Zoxc.

bors · 2019-12-22T13:33:59Z

☀️ Test successful - checks-azure
Approved by: eddyb
Pushing 26286c7 to master...

Rustup to rust-lang/rust#66931 changelog: none

Allocate HIR on an arena 2/4 -- Expr & Pat This is the second PR in the series started by #66931 This time, commits don't really make sense on their own. They are mostly split by type of compile error. The additional diff is here: cjgillot/rust@hirene-preamble...hirene-expr

Allocate HIR on an arena 3/4 -- Ty This is the third PR in the series started by #66931 and #66936 Once again, commits don't really make sense on their own. They are mostly split by type of compile error. The additional diff is here: cjgillot/rust@hirene-expr...hirene-ty

@eddyb

Allocate HIR on an arena 4/4 This is the fourth and last PR in the series started by #66931, #66936 and #66942. The last commits should compile on their own. The difference with the previous PR is given by cjgillot/rust@hirene-ty...hirene A few more cleanups may be necessary, please tell me. r? @eddyb like the other cc @Zoxc

rust-highfive assigned matthewjasper Dec 1, 2019

rust-highfive assigned eddyb and unassigned matthewjasper Dec 1, 2019

Zoxc reviewed Dec 1, 2019

View reviewed changes

Centril reviewed Dec 1, 2019

View reviewed changes

src/librustc/hir/lowering.rs Show resolved Hide resolved

Zoxc reviewed Dec 1, 2019

View reviewed changes

Centril reviewed Dec 1, 2019

View reviewed changes

Zoxc reviewed Dec 1, 2019

View reviewed changes

This was referenced Dec 1, 2019

Allocate HIR on an arena 2/4 -- Expr & Pat #66936

Merged

Allocate HIR on an arena 3/4 -- Ty #66942

Merged

bors added the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Dec 2, 2019

cjgillot force-pushed the hirene-preamble branch from e2ad73e to 8985208 Compare December 2, 2019 19:40

cjgillot force-pushed the hirene-preamble branch from bf0c6fb to 15b76ce Compare December 3, 2019 22:18

cjgillot mentioned this pull request Dec 4, 2019

Allocate HIR on an arena 4/4 #67032

Merged

eddyb approved these changes Dec 5, 2019

View reviewed changes

cjgillot force-pushed the hirene-preamble branch from 15b76ce to 0c4b8ee Compare December 9, 2019 12:18

cjgillot added 5 commits December 21, 2019 23:33

Use Arena inside hir::StructField.

e252612

Use Arena inside hir::Mod.

42c03e4

Use Arena inside hir::Body.

9694ab9

Tidy.

5f5b6e7

Nits.

baa49b2

cjgillot force-pushed the hirene-preamble branch from f5ce0e7 to baa49b2 Compare December 21, 2019 22:42

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Dec 22, 2019

bors added the merged-by-bors This PR was explicitly merged by bors. label Dec 22, 2019

bors merged commit baa49b2 into rust-lang:master Dec 22, 2019

This was referenced Dec 22, 2019

Cleanup rmeta::MacroDef #66364

Merged

Don't warn about snake case for field puns. #66660

Merged

matthiaskrgr mentioned this pull request Dec 22, 2019

toolstate not updated rust-lang-nursery/rust-toolstate#15

Closed

flip1995 added a commit to flip1995/rust-clippy that referenced this pull request Dec 22, 2019

Rustup to rust-lang/rust#66931

f6a5b60

bors added a commit to rust-lang/rust-clippy that referenced this pull request Dec 22, 2019

Auto merge of #4938 - flip1995:rustup, r=flip1995

6d12259

Rustup to rust-lang/rust#66931 changelog: none

cjgillot deleted the hirene-preamble branch December 22, 2019 17:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allocate HIR on an arena 1/4 #66931

Allocate HIR on an arena 1/4 #66931

cjgillot commented Dec 1, 2019

matthewjasper commented Dec 1, 2019

Zoxc commented Dec 1, 2019

Zoxc Dec 1, 2019

Zoxc Dec 1, 2019

cjgillot Dec 1, 2019

Zoxc Dec 1, 2019

Zoxc Dec 1, 2019

Zoxc Dec 1, 2019

Centril Dec 1, 2019

Zoxc Dec 1, 2019

Zoxc Dec 1, 2019

cjgillot Dec 1, 2019

cjgillot Dec 11, 2019

Zoxc Dec 1, 2019

Zoxc Dec 1, 2019

Zoxc Dec 1, 2019

Zoxc Dec 1, 2019

Zoxc Dec 1, 2019

Zoxc Dec 1, 2019

Zoxc Dec 1, 2019

bors commented Dec 2, 2019

cjgillot commented Dec 3, 2019

eddyb left a comment

bors commented Dec 9, 2019

nikomatsakis commented Dec 9, 2019

bors commented Dec 9, 2019

cjgillot commented Dec 21, 2019

Zoxc commented Dec 22, 2019

Zoxc commented Dec 22, 2019

bors commented Dec 22, 2019

bors commented Dec 22, 2019

matthewjasper commented Dec 22, 2019

bors commented Dec 22, 2019

bors commented Dec 22, 2019

Allocate HIR on an arena 1/4 #66931

Allocate HIR on an arena 1/4 #66931

Conversation

cjgillot commented Dec 1, 2019

matthewjasper commented Dec 1, 2019

Zoxc commented Dec 1, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bors commented Dec 2, 2019

cjgillot commented Dec 3, 2019

eddyb left a comment

Choose a reason for hiding this comment

bors commented Dec 9, 2019

nikomatsakis commented Dec 9, 2019

bors commented Dec 9, 2019

cjgillot commented Dec 21, 2019

Zoxc commented Dec 22, 2019

Zoxc commented Dec 22, 2019

bors commented Dec 22, 2019

bors commented Dec 22, 2019

matthewjasper commented Dec 22, 2019

bors commented Dec 22, 2019

bors commented Dec 22, 2019