Explain stages (take N) #843

jyn514 · 2020-09-01T01:10:46Z

Successor to #807 that doesn't try to change what 'stage' means. Instead, it accepts that there are two meanings and explains both, as well as the context in which they are used.

Before reading the text, I strongly recommend taking a look at the diagram added (thanks to @mark-i-m for writing it up!).

I would like to do the following before this merges:

Renamed link-stage to build-stage in the diagram. While it's true that those stages are linked together, 'build' more accurately describes how x.py refers to them. I'm willing to be convinced the opposite direction for this, in which case build-stage should be renamed to link-stage in the text. Done.
Add the directories in build/*/stage* corresponding to each cell of the diagram - I haven't done this, but it's discussed in detail in 'Understanding Stages of Bootstrap', so maybe it's not necessary?

~~I don't have access to the TeX sources for the diagram, so both of the above are waiting on @mark-i-m (no pressure or anything :P).~~ Mark got me the .tex file (thank you!)

I'm also interested in cleaning up the existing 'Stage of Boostrap' section, but I'm not exactly sure how best to do that ... there's definitely good information there, it's just hard to find in the wall of text.

Comments welcome!
cc @eddyb @Mark-Simulacrum

jyn514 · 2020-09-01T01:11:49Z

The diagram:

jyn514 · 2020-09-01T01:22:08Z

More of the discussion that came up with this (warning: lots and lots of backlog): https://rust-lang.zulipchat.com/#narrow/stream/196385-t-compiler.2Fwg-rustc-dev-guide/topic/meeting.2008.2E04.2E2020/near/205939628

jyn514 · 2020-09-01T02:08:07Z

src/building/bootstrapping.md

+
+<!-- TODO: label each of the cells with the name of the directory in `build` it corresponds to -->
+
+![stages](./stages.png)


@RDambrosio016 mentioned that I should probably explain what stage 3 is and that it's not really useful except for verifying reproducible builds.

RalfJung · 2020-09-01T09:18:33Z

@jyn514 the diagram says that x.py build --stage 1 would correspond to column 1, but is that true? AFAIK with the latest planned changes, that would also build the "stage 1 std artifacts", i.e. it would build all of row 1. Which also makes much more sense when compared with x.py test --stage 1.

jyn514 · 2020-09-01T12:39:44Z

@jyn514 the diagram says that x.py build --stage 1 would correspond to column 1, but is that true? AFAIK with the latest planned changes, that would also build the "stage 1 std artifacts", i.e. it would build all of row 1. Which also makes much more sense when compared with x.py test --stage 1.

It's worded poorly in the diagram - build --stage 1 corresponds to building something in column 1, but since rust-lang/compiler-team#326 that something is just libstd, not the compiler. However if you run build --stage 1 src/rustc, it will build the rustc in column 1, not the rustc in row 1.

jyn514 · 2020-09-01T12:41:09Z

Or in other words, build --stage 1 src/rustc does more work than build --stage 1, not less.

spastorino · 2020-09-01T14:12:36Z

I like this ❤️

RalfJung · 2020-09-01T14:15:01Z

However if you run build --stage 1 src/rustc, it will build the rustc in column 1, not the rustc in row 1.

Oh, that's what you mean.

I am just wondering why it seems so inconsistent here (rows vs columns), when it doesn't feel so inconsistent when I use it.

RalfJung · 2020-09-01T14:23:23Z

The way I think about it is that x.py build --stage N defines the compiler that is used to build this thing (that matches the column view), and x.py test --stage N defines the compiler that is tested. That needs a matching std, so I'd have a "tests" box between "std" and "rustc", and then I'd say it refers to running the tests in the given column. That makes it always columns.

Also, should "tools" (clippy, miri) be included in the diagram? The compiler-dependent tools would be below rustc. (The other tools are not subject to staging I think.)

jyn514 · 2020-09-01T15:51:48Z

x.py build --stage N defines the compiler that is used to build this thing (that matches the column view), and x.py test --stage N defines the compiler that is tested

That corresponds to the columns/rows view, no? Where tested corresponds to 'run-stage'?

That needs a matching std, so I'd have a "tests" box between "std" and "rustc", and then I'd say it refers to running the tests in the given column. That makes it always columns.

Hmm ... I'd put it in the same cell as std I think, because it's not on a stage boundary.

But I don't think your description is accurate: when you say test --stage 1 that builds one rustc, not two of them. So that means that it's the row view.

Also, should "tools" (clippy, miri) be included in the diagram? The compiler-dependent tools would be below rustc. (The other tools are not subject to staging I think.)

Good idea, I'll do that. They should go in the same cell as rustc I think.

jyn514 · 2020-09-01T15:52:11Z

(except rustdoc because rustdoc is special and it would just be even more confusing)

mark-i-m · 2020-09-01T19:12:05Z

I posted on zulip too, but just so it's easy to find, here is the tex of the diagram: https://gist.github.com/mark-i-m/e0c885f7b585f02232962b5be0d457f8

workingjubilee · 2020-09-02T00:42:17Z

...OH.

RalfJung · 2020-09-02T11:27:59Z

But I don't think your description is accurate: when you say test --stage 1 that builds one rustc, not two of them. So that means that it's the row view.

I think I have a consistent picture that never uses the "row view" so do I don't think you need that view.

With there being a "test" in between "std" and "rustc" (or in the same cell as "std"), then x.py test --stage 1 would select the "test" node in column 1 and run that (and build its dependencies first). This accurately explains why only one rustc is built.

jyn514 · 2020-09-02T14:35:39Z

I had a whole speech about how I thought my viewpoint was right, but after thinking about it now I'm not sure any more ... the way you see builds is that build --stage N and test --stage N always runs stageN/rustc? And build --stage N src/rustc is consistent because it still runs stageN/?

I guess that implies the current system for doc is consistent, since it runs stageN/rustdoc, but that build --stage N src/tools/rustdoc is inconsistent because it runs stage(N-1)/rustc.

@eddyb what do you think about this?

jyn514 · 2020-09-02T14:39:07Z

and to clear up some confusion on my end: when I build --stage 1 library/std, is that running stage1/rustc or stage0/rustc?

Mark-Simulacrum · 2020-09-02T14:41:04Z

I find that question confusing because everything runs stage0/bin/rustc. But it finishes up by buillding the standard library with stage1/bin/rustc

jyn514 · 2020-09-02T14:41:40Z

it finishes up by buillding the standard library with stage1/bin/rustc

Ok yes that was my question, thanks. So that's consistent with the model @RalfJung has.

RalfJung · 2020-09-03T12:27:53Z

I think there's probably multiple "right" viewpoints that explain the same behavior in different ways. I just wonder which one is "best" in the sense of being easiest to teach and learn.

workingjubilee · 2020-09-04T08:28:16Z

I did not really understand it until I looked at the diagram presented here, in spite of reading the description in rustc-dev-guide several times, and I have repeatedly tried to run x.py only to have it fail (including "spend 1 hour and then fail", or do something unexpected) for completely-mysterious-to-me-at-the-time reasons that are now obviously having to do with the compiler staging not following a model I expected. So I would expect the amount of time recovered by this visual depiction alone would be significant. And I've seen the other visual flowcharts, but they were not as clear to me (because they were more complicated, trying to follow the exact execution and depicting all possible details instead of focusing on the main thread).

The words around it are mostly secondary, imo.

jyn514 · 2020-09-04T12:31:33Z

Maybe it would be helpful to document both models of staging? Mention that the guide and various other places use the '--stage N runs stageN/rustc' model, but still have this diagram?

infinity0 · 2020-09-07T10:54:44Z

@jyn514 The diagram is great. However, as I understand from the latest changes, the wording needs to be amended:

build --stage N $X means build $X from column N
test --stage N $X means (build then) test $X from row N
doc --stage N $X means (build then) document $X from row N

If $X is omitted, this implies $X = everything in row N.

jyn514 · 2020-09-07T13:38:15Z

Yes to all points except

If $X is omitted, this implies $X = everything in row N.

Right now tools are built as well, I want to change that though: rust-lang/compiler-team#351

infinity0 · 2020-09-07T21:28:49Z

It would be nice to clarify what happens on install and also on cross-build as well. For example with a regular non-cross build it is sufficient to install rustc+std in row 2, however in a cross-build there is an additional rustc:foreign-arch being built at row 2 column 1, which is not used to build the std at row 2 column 2, but would be used during the install --stage 2 step. (I hope I got that right, kind of guessing here.)

In my head I can imagine representing this with another dimension to the diagram but that might not look so nice in practice.

jyn514 · 2020-09-07T23:28:01Z

I would need help with cross-compiling - maybe that could be a follow-up PR? Definitely good information to have, though.

jyn514 · 2020-09-08T01:42:04Z

I started by editing the diagram heavily, then decided I didn't like it and just made minor tweaks. How does this look?

jyn514 · 2020-09-08T01:48:44Z

Also, should "tools" (clippy, miri) be included in the diagram? The compiler-dependent tools would be below rustc. (The other tools are not subject to staging I think.)

I put these in prose instead, since they always match up with the rustc boxes. The full order of dependencies (core -> alloc -> std -> rustc -> {rustdoc, clippy, miri}) are shown below in the 'Stages of Bootstrap' section so I didn't put them in the diagram - I want to avoid making it too crowded.

I also added a section for @RalfJung's view of 'stage' as 'run stageN/rustc' - I don't know that it's the first model I want to teach to newcomers, but it's definitely valid and might help people confused by the diagram.

@RDambrosio016 mentioned that I should probably explain what stage 3 is and that it's not really useful except for verifying reproducible builds.

This was added in #851, I didn't discuss it more here.

Can't change this in the diagram until I find the sources.

ehuss · 2020-09-08T14:54:31Z

Just some feedback on the overall direction here, I feel like this makes it more complicated and confusing. Adding multiple new terminology (run-stage and build-stage) that haven't been used before seems to add layers of complexity that I don't really understand. I also think it would not be a good idea to try to introduce multiple mental models.

I appreciate trying to improve the documentation, and am thankful you are trying to tackle it. I think the wall of text on bootstrapping is too much, but this seems to add more content, instead of remove it. I guess I am a bit uncertain why people are so confused by the topic, as it isn't that complicated. My personal confusion in the past mostly stemmed around feeling like the numbering seemed off-by-one (like, the "stage 0" compiler created a "stage0-rustc" directory, which seemed confusing because I expected it to create stage N+1). But at the time there wasn't much documentation, and this was just me fumbling around. Once I learned "stage" was more of a "step in a process", and the relationship between the --stage flag and the path specified on the command-line, things started to click better. I still feel like it could be improved somehow, though I'm not sure how. But in terms of documentation, I think it would be good to simplify it, not add more to it.

camelid

Overall, it looks like it's off to a good start, but I still find some pieces confusing. I think they just need a bit more explanation to give more context to people who are newer to rustc (like me!). The part that I feel still needs the most explanation is the distinction between run-stage and build-stage.

I left a bunch of comments; I hope they're not overwhelming! Some of them are just small typographical things though.

Let me know if you need more clarification about my comments! And also, thank you for putting in the work to explain this better in the guide!

camelid · 2020-09-08T19:07:44Z

src/building/bootstrapping.md

+Does `--stage 1` mean to _build_ the stage 1 artifacts or to _run_ them?
+In fact, it means both!
+
+<!-- TODO: label each of the cells with the name of the directory in `build` it corresponds to -->


Is this TODO still valid?

camelid · 2020-09-08T19:10:53Z

src/building/stages.tex

+Rows are things that can be used together to build a program.
+\\
+
+\noindent \verb|./x.py build --stage N| builds programs in \emph{column} \verb|N| (except rustdoc - it builds rustdoc from column \verb|N - 1|).\\


To get a proper en dash:

Suggested change

\noindent \verb|./x.py build --stage N| builds programs in \emph{column} \verb|N| (except rustdoc - it builds rustdoc from column \verb|N - 1|).\\

\noindent \verb|./x.py build --stage N| builds programs in \emph{column} \verb|N| (except rustdoc -- it builds rustdoc from column \verb|N - 1|).\\

camelid · 2020-09-08T19:13:55Z

src/building/bootstrapping.md

+- `rustfmt`
+- `clippy`
+- `miri`
+- compiler plugins


What exactly is meant by "compiler plugins"? When I hear "compiler plugins" I think of procedural macros, but I don't think that's what's meant here.

camelid · 2020-09-08T19:14:57Z

src/building/bootstrapping.md

+### What are `run-stage` and `build-stage`?
+
+`run-stage` means that this deals with _running_ the compiler,
+so `--stage N` refers to the artifacts in `build/stageN`.


Same as below: I think it's clearer if you delineate what is literal text and what is filled in with some value.

Suggested change

so `--stage N` refers to the artifacts in `build/stageN`.

so `--stage N` refers to _running_ the artifacts in `build/<toolchain>/stageN`.

camelid · 2020-09-08T19:16:41Z

src/building/bootstrapping.md

+so `--stage N` refers to the artifacts in `build/stageN`.
+
+`build-stage` means that this deals with _building_ the compiler,
+and it refers to `build/stageN-component`.


I think it's clearer if you delineate what is literal text and what is filled in with some value.

Suggested change

and it refers to `build/stageN-component`.

and it refers to `build/<toolchain>/stageN-<component>`.

camelid · 2020-09-08T20:38:05Z

src/building/bootstrapping.md

+- `test`
+- `proc_macro`
+
+### What are `run-stage` and `build-stage`?


I still don't feel like I fully understand what the distinction between these is.

camelid · 2020-09-08T20:39:23Z

src/building/stages.tex

+\noindent \verb|build| excludes \verb|rustc| by default.
+
+Use \verb|build --stage N src/rustc| to build the \verb|rustc| in \emph{column}
+\verb|N|.


This confuses me. It sounds like it means that ./x.py build doesn't build the compiler, but that doesn't seem right.

camelid · 2020-09-08T20:39:33Z

src/building/stages.tex

+\node[text width=5in] at (2.5, -6) {
+\noindent \verb|build| excludes \verb|rustc| by default.
+
+Use \verb|build --stage N src/rustc| to build the \verb|rustc| in \emph{column}


Suggested change

Use \verb|build --stage N src/rustc| to build the \verb|rustc| in \emph{column}

Use \verb|build --stage N compiler/rustc| to build the \verb|rustc| in \emph{column}

camelid · 2020-09-08T20:40:44Z

src/building/bootstrapping.md

+
+### Why have `build-stage` at all?
+
+`stage0/bin/rustc` can't open an rlib from stage1-* or vice-versa.


Suggested change

`stage0/bin/rustc` can't open an rlib from stage1-* or vice-versa.

`stage0/bin/rustc` can't open an rlib (a Rust static library archive) from `stage1-*` or vice-versa.

camelid · 2020-09-08T20:44:34Z

src/building/bootstrapping.md

@@ -276,6 +346,8 @@ artifacts respectively.
 Additionally, the `RUSTFLAGS_STAGE_NOT_0` variable, as its name suggests, pass


This is not from this PR, but I figure we might as well fix it now!

Suggested change

Additionally, the `RUSTFLAGS_STAGE_NOT_0` variable, as its name suggests, pass

Additionally, the `RUSTFLAGS_STAGE_NOT_0` variable, as its name suggests, passes

jyn514 · 2020-09-08T21:43:57Z

Closing this in favor of #857.

jyn514 added help wanted Extra attention is needed waiting-on-review This PR is waiting for a reviewer to verify its content labels Sep 1, 2020

This was referenced Sep 1, 2020

x.py's naming of stages is confusing rust-lang/rust#59864

Open

Fix off-by-one-error in stage numbering between rustc-guide and rustbuild rust-lang/rust#57963

Closed

jyn514 mentioned this pull request Sep 1, 2020

Add --run-stage and --link-stage aliases rust-lang/rust#75167

Closed

jyn514 commented Sep 1, 2020

View reviewed changes

jyn514 mentioned this pull request Sep 5, 2020

Make the default stage for x.py configurable rust-lang/rust#76165

Closed

jyn514 added waiting-on-author This PR is waiting for additional action by the OP and removed waiting-on-review This PR is waiting for a reviewer to verify its content labels Sep 7, 2020

jyn514 added 8 commits September 7, 2020 21:49

Use _both_ models for stages

c0bd637

Mention that the text in Stages of Bootstrap refers to link-stage

13772e6

Steal eddyb's words to say why link-stage is useful

0c3ec37

Use build-stage consistently

068bcda

Can't change this in the diagram until I find the sources.

line length

caac2dc

Tweak with the diagram a little more

e4d68e8

Mention RalfJung's view of stages

afc7c65

Rustdoc from stage N *minus* 1

6db155d

jyn514 force-pushed the 2d-stages branch from 69fa00a to 6db155d Compare September 8, 2020 01:49

camelid requested changes Sep 8, 2020

View reviewed changes

jyn514 mentioned this pull request Sep 8, 2020

Explain stages in terms of the compiler currently running (take N+1) #857

Merged

jyn514 closed this Sep 8, 2020

jyn514 mentioned this pull request Oct 6, 2020

Find a way to incorporate the 2D diagram into the section about bootstrapping #914

Closed

jyn514 mentioned this pull request Nov 30, 2020

Allow building rustdoc without first building rustc (MVP) rust-lang/rust#79540

Merged

jyn514 deleted the 2d-stages branch January 29, 2021 04:13

jyn514 mentioned this pull request Aug 8, 2022

Add the diagrams from my rustconf talk #1419

Closed

RalfJung mentioned this pull request Mar 31, 2024

complicated build stages rust-lang/rust#119946

Open


		<!-- TODO: label each of the cells with the name of the directory in `build` it corresponds to -->

		![stages](./stages.png)

	\noindent \verb\|./x.py build --stage N\| builds programs in \emph{column} \verb\|N\| (except rustdoc - it builds rustdoc from column \verb\|N - 1\|).\\
	\noindent \verb\|./x.py build --stage N\| builds programs in \emph{column} \verb\|N\| (except rustdoc -- it builds rustdoc from column \verb\|N - 1\|).\\

	so `--stage N` refers to the artifacts in `build/stageN`.
	so `--stage N` refers to _running_ the artifacts in `build/<toolchain>/stageN`.

	and it refers to `build/stageN-component`.
	and it refers to `build/<toolchain>/stageN-<component>`.

	Use \verb\|build --stage N src/rustc\| to build the \verb\|rustc\| in \emph{column}
	Use \verb\|build --stage N compiler/rustc\| to build the \verb\|rustc\| in \emph{column}


		### Why have `build-stage` at all?

		`stage0/bin/rustc` can't open an rlib from stage1-* or vice-versa.

	`stage0/bin/rustc` can't open an rlib from stage1-* or vice-versa.
	`stage0/bin/rustc` can't open an rlib (a Rust static library archive) from `stage1-*` or vice-versa.

		@@ -276,6 +346,8 @@ artifacts respectively.
		Additionally, the `RUSTFLAGS_STAGE_NOT_0` variable, as its name suggests, pass

Explain stages (take N) #843

Explain stages (take N) #843

Conversation

jyn514 commented Sep 1, 2020 • edited Loading

jyn514 commented Sep 1, 2020

jyn514 commented Sep 1, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RalfJung commented Sep 1, 2020

jyn514 commented Sep 1, 2020

jyn514 commented Sep 1, 2020

spastorino commented Sep 1, 2020

RalfJung commented Sep 1, 2020

RalfJung commented Sep 1, 2020 • edited Loading

jyn514 commented Sep 1, 2020

jyn514 commented Sep 1, 2020

mark-i-m commented Sep 1, 2020

workingjubilee commented Sep 2, 2020

RalfJung commented Sep 2, 2020 • edited Loading

jyn514 commented Sep 2, 2020

jyn514 commented Sep 2, 2020

Mark-Simulacrum commented Sep 2, 2020

jyn514 commented Sep 2, 2020 • edited Loading

RalfJung commented Sep 3, 2020

workingjubilee commented Sep 4, 2020

jyn514 commented Sep 4, 2020

infinity0 commented Sep 7, 2020

jyn514 commented Sep 7, 2020

infinity0 commented Sep 7, 2020

jyn514 commented Sep 7, 2020

jyn514 commented Sep 8, 2020 • edited Loading

jyn514 commented Sep 8, 2020

ehuss commented Sep 8, 2020

camelid left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jyn514 commented Sep 8, 2020

jyn514 commented Sep 1, 2020 •

edited

Loading

RalfJung commented Sep 1, 2020 •

edited

Loading

RalfJung commented Sep 2, 2020 •

edited

Loading

jyn514 commented Sep 2, 2020 •

edited

Loading

jyn514 commented Sep 8, 2020 •

edited

Loading