Explain stages (take N)

README.md

@@ -76,6 +76,24 @@ and execute the following command in the root of the repository:
 
 The build files are found in the `book` directory.
 
+### Regenerating `stages.png`
+
+This requires a LaTeX toolchain installed.
+On Debian-based distros you can install it with
+```
+$ sudo apt install texlive-latex-base
+```
+
+Then run
+```
+$ pdflatex -output-directory=target src/building/stages.tex
+```
+
+and take a screenshot of the resulting PDF in your browser:
+```
+$ x-www-browser target/stages.pdf
+```
+
 ### Link Validations
 
 We use `mdbook-linkcheck` to validate URLs included in our documentation. To perform link checks, uncomment the `[output.linkcheck]` field in the `book.toml` configuration file and install `mdbook-linkcheck` with:

src/building/bootstrapping.md

@@ -57,6 +57,75 @@ first build the new compiler with an older compiler and then use that to
 build the new compiler with itself. For development, you usually only want
 the `stage1` compiler: `x.py build library/std`.
 
+## Where do stages start and end?
+
+A common question is what exactly happens when you run `x.py build` or `x.py test`.
+Does `--stage 1` mean to _build_ the stage 1 artifacts or to _run_ them?
+In fact, it means both!
+
+<!-- TODO: label each of the cells with the name of the directory in `build` it corresponds to -->
+
+![stages](./stages.png)
+
+So, for example, when you run `x.py test [--stage 1]`,
+that means to build the compiler in row 1 and column 0, then run it on the testsuite.
+This corresponds to the `run-stage` diagram.
+However, when you run `x.py build [--stage 1]`, that means to build the compiler in
+row 2 and column 1. This corresponds to the `build-stage` diagram.
+Building any of the items in the diagram also requires first building all
+items with arrows pointing to it.
+
+The diagram just says `rustc` for simplicity, but this also includes all
+programs linked to rustc:
+
+- `rustdoc`
+- `rustfmt`
+- `clippy`
+- `miri`
+- compiler plugins
+
+Similarly, `std` refers to the whole standard library:
+
+- `core`
+- `alloc`
+- `std`
+- `test`
+- `proc_macro`
+
+### What are `run-stage` and `build-stage`?
+
+`run-stage` means that this deals with _running_ the compiler,
+so `--stage N` refers to the artifacts in `build/stageN`.
+
+`build-stage` means that this deals with _building_ the compiler,
+and it refers to `build/stageN-component`.
+
+`build/stageN` is suitable for use with `rustup toolchain link`,
+but `stageN-component` never has enough components to be usable (since it only has one).
+Copying these artifacts from `stage(N-1)-component` to `stageN`
+is called _uplifting_ the artifacts to `stageN`.
+
+### Why have `build-stage` at all?
+
+`stage0/bin/rustc` can't open an rlib from stage1-* or vice-versa.
+They are completely separate worlds, and `build-stage` reflects those worlds quite directly.
+Say you want to build a custom driver and you've run
+`rustup toolchain link build/*/stage1`: you have to run
+`x.py build --stage 1 src/librustc_driver` to have it available.
+The stage number corresponds to the "world" you have to be in to use it.
+If this used `run-stage` instead, you'd need `x.py build --stage 1` to build
+a regular program, but `x.py build --stage 2 src/librustc_driver` to build a
+custom driver.
+
+### Are there other ways to think about stages?
+
+Yes! The above explanation focuses on what `rustc` is being _referred_ to -
+it describes `build --stage 1 src/rustc` as `build-stage` 1 and `run-stage 2`.
+However, another way to think about it is that `--stage 1` refers to
+the compiler _being run_, so `build --stage 1 src/rustc` means to _run_
+`stage1/rustc` on the `src/rustc` crate. This can be slightly more confusing
+at first, but leads to a more consistent view of 'stage'.
+
 ## Complications of bootstrapping
 
 Since the build system uses the current beta compiler to build the stage-1
@@ -152,7 +221,8 @@ Keep in mind this diagram is a simplification, i.e. `rustdoc` can be built at
 different stages, the process is a bit different when passing flags such as
 `--keep-stage`, or if there are non-host targets.
 
-The following tables indicate the outputs of various stage actions:
+The following tables indicate the outputs of various stage actions
+(in this context, 'stage' refers to `build-stage`):
 
 | Stage 0 Action                                            | Output                                       |
 |-----------------------------------------------------------|----------------------------------------------|
@@ -192,7 +262,7 @@ The following tables indicate the outputs of various stage actions:
 
 `--stage=2` stops here.
 
-Note that the convention `x.py` uses is that:
+Note that the convention `x.py` uses for `build-stage` is that:
 - A "stage N artifact" is an artifact that is _produced_ by the stage N compiler.
 - The "stage (N+1) compiler" is assembled from "stage N artifacts".
 - A `--stage N` flag means build _with_ stage N.
@@ -276,6 +346,8 @@ artifacts respectively.
 Additionally, the `RUSTFLAGS_STAGE_NOT_0` variable, as its name suggests, pass
 the given arguments if the stage is not 0.
 
+In this context, `STAGE` refers to `build-stage`.
+
 ## Environment Variables
 
 During bootstrapping, there are a bunch of compiler-internal environment

src/building/stages.tex

@@ -0,0 +1,79 @@
+\documentclass{standalone}
+
+\usepackage{tikz}
+\usetikzlibrary{arrows.meta}
+
+\begin{document}
+
+\noindent
+\begin{tikzpicture}
+
+\node[text width=5in] at (2.5, 2) {
+\noindent Arrows represent build dependencies.
+Columns are things that are linked together.
+Rows are things that can be used together to build a program.
+\\
+
+\noindent \verb|./x.py build --stage N| builds programs in \emph{column} \verb|N| (except rustdoc - it builds rustdoc from column \verb|N - 1|).\\
+\noindent \verb|./x.py test --stage N| builds and tests everything in \emph{row} \verb|N|.\\
+\noindent \verb|./x.py doc --stage N| generates docs with rustdoc from \emph{row} \verb|N|.\\
+};
+
+\draw[draw=black,fill=white,fill opacity=0.5] (-3, -0.5) rectangle ++(12, 1);
+\draw[fill=teal,fill opacity=0.5] (-3, -1.5) rectangle ++(12, 1);
+\draw[fill=olive,fill opacity=0.5] (-3, -2.5) rectangle ++(12, 1);
+\draw[fill=pink,fill opacity=0.5] (-3, -3.5) rectangle ++(12, 1);
+
+\draw[draw=black,fill=white,fill opacity=0.5] (1, 0.5) rectangle ++(2, -5);
+\draw[fill=teal,fill opacity=0.5] (3, 0.5) rectangle ++(2, -5);
+\draw[fill=olive,fill opacity=0.5] (5, 0.5) rectangle ++(2, -5);
+\draw[fill=pink,fill opacity=0.5] (7, 0.5) rectangle ++(2, -5);
+
+\node[rotate=90] at (-3.5, -1.5) {run-stage};
+
+\node[] at (-2, 0) {stage 0};
+\node[] at (-2, -1) {stage 1};
+\node[] at (-2, -2) {stage 2};
+\node[] at (-2, -3) {stage 3};
+
+\node[] at (5, -5) {build-stage};
+
+\node[] at (2, -4) {stage 0};
+\node[] at (4, -4) {stage 1};
+\node[] at (6, -4) {stage 2};
+\node[] at (8, -4) {stage 3};
+
+\begin{scope}[every node/.style={thick,draw,fill=white}]
+	\node (s0r) at (0,0) {beta rustc};
+	\node (s0s) at (2,0) {std};
+	\node (s1r) at (2,-1) {rustc};
+	\node (s1s) at (4,-1) {std};
+	\node (s2r) at (4,-2) {rustc};
+	\node (s2s) at (6,-2) {std};
+	\node (s3r) at (6,-3) {rustc};
+	\node (s3s) at (8,-3) {std};
+\end{scope}
+
+\begin{scope}[>={Stealth[black]}, every edge/.style={draw=black,very thick}]
+	\path [->] (s0r) edge node {} (s0s);
+	\path [->] (s0r) edge node {} (s1r);
+	\path [->] (s0s) edge node {} (s1r);
+	\path [->] (s1r) edge node {} (s1s);
+	\path [->] (s1r) edge node {} (s2r);
+	\path [->] (s1s) edge node {} (s2r);
+	\path [->] (s2r) edge node {} (s2s);
+	\path [->] (s2r) edge node {} (s3r);
+	\path [->] (s2s) edge node {} (s3r);
+	\path [->] (s3r) edge node {} (s3s);
+\end{scope}
+
+\node[text width=5in] at (2.5, -6) {
+\noindent \verb|build| excludes \verb|rustc| by default.
+
+Use \verb|build --stage N src/rustc| to build the \verb|rustc| in \emph{column}
+\verb|N|.
+};
+
+\end{tikzpicture}
+
+\end{document}