From 4d40ef4ef4acd411d164d910319a22c70867eeda Mon Sep 17 00:00:00 2001 From: Eric Huss Date: Fri, 18 Sep 2020 13:17:58 -0700 Subject: [PATCH 1/3] Add contributor guide. --- .github/ISSUE_TEMPLATE/tracking_issue.md | 34 +++ .github/workflows/contrib.yml | 33 +++ ARCHITECTURE.md | 150 ------------ CONTRIBUTING.md | 231 +----------------- README.md | 6 +- crates/cargo-test-support/src/lib.rs | 109 +-------- src/doc/contrib/README.md | 12 + src/doc/contrib/book.toml | 3 + src/doc/contrib/src/SUMMARY.md | 20 ++ src/doc/contrib/src/architecture/codebase.md | 72 ++++++ .../contrib/src/architecture/compilation.md | 39 +++ src/doc/contrib/src/architecture/console.md | 82 +++++++ src/doc/contrib/src/architecture/files.md | 67 +++++ src/doc/contrib/src/architecture/index.md | 1 + src/doc/contrib/src/architecture/packages.md | 92 +++++++ .../contrib/src/architecture/subcommands.md | 25 ++ src/doc/contrib/src/design.md | 101 ++++++++ src/doc/contrib/src/index.md | 29 +++ src/doc/contrib/src/issues.md | 109 +++++++++ src/doc/contrib/src/process/index.md | 124 ++++++++++ src/doc/contrib/src/process/release.md | 109 +++++++++ src/doc/contrib/src/process/unstable.md | 83 +++++++ .../contrib/src/process/working-on-cargo.md | 147 +++++++++++ src/doc/contrib/src/tests/index.md | 20 ++ src/doc/contrib/src/tests/profiling.md | 34 +++ src/doc/contrib/src/tests/running.md | 41 ++++ src/doc/contrib/src/tests/writing.md | 132 ++++++++++ 27 files changed, 1418 insertions(+), 487 deletions(-) create mode 100644 .github/ISSUE_TEMPLATE/tracking_issue.md create mode 100644 .github/workflows/contrib.yml delete mode 100644 ARCHITECTURE.md create mode 100644 src/doc/contrib/README.md create mode 100644 src/doc/contrib/book.toml create mode 100644 src/doc/contrib/src/SUMMARY.md create mode 100644 src/doc/contrib/src/architecture/codebase.md create mode 100644 src/doc/contrib/src/architecture/compilation.md create mode 100644 src/doc/contrib/src/architecture/console.md create mode 100644 src/doc/contrib/src/architecture/files.md create mode 100644 src/doc/contrib/src/architecture/index.md create mode 100644 src/doc/contrib/src/architecture/packages.md create mode 100644 src/doc/contrib/src/architecture/subcommands.md create mode 100644 src/doc/contrib/src/design.md create mode 100644 src/doc/contrib/src/index.md create mode 100644 src/doc/contrib/src/issues.md create mode 100644 src/doc/contrib/src/process/index.md create mode 100644 src/doc/contrib/src/process/release.md create mode 100644 src/doc/contrib/src/process/unstable.md create mode 100644 src/doc/contrib/src/process/working-on-cargo.md create mode 100644 src/doc/contrib/src/tests/index.md create mode 100644 src/doc/contrib/src/tests/profiling.md create mode 100644 src/doc/contrib/src/tests/running.md create mode 100644 src/doc/contrib/src/tests/writing.md diff --git a/.github/ISSUE_TEMPLATE/tracking_issue.md b/.github/ISSUE_TEMPLATE/tracking_issue.md new file mode 100644 index 00000000000..7115200e10a --- /dev/null +++ b/.github/ISSUE_TEMPLATE/tracking_issue.md @@ -0,0 +1,34 @@ +--- +name: Tracking Issue +about: A tracking issue for an accepted feature or RFC in Cargo. +title: Tracking Issue for XXX +labels: C-tracking-issue +--- + + +**About tracking issues** + +Tracking issues are used to record the overall progress of implementation. +They are also used as hubs connecting to other relevant issues, e.g., bugs or open design questions. +A tracking issue is however *not* meant for large scale discussion, questions, or bug reports about a feature. +Instead, open a dedicated issue for the specific matter and add the relevant feature gate label. + +**Summary** + +Original issue: #NNNN +Implementation: #NNNN +Documentation: https://doc.rust-lang.org/nightly/cargo/reference/unstable.html#my-feature + + + +**Unresolved issues** + +* [ ] Make a list of any known implementation or design issues. + +**Future extensions** + + diff --git a/.github/workflows/contrib.yml b/.github/workflows/contrib.yml new file mode 100644 index 00000000000..4c0a36e3374 --- /dev/null +++ b/.github/workflows/contrib.yml @@ -0,0 +1,33 @@ +name: Contrib Deploy +on: + push: + branches: + - master + +jobs: + deploy: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v2 + with: + fetch-depth: 0 + - name: Install mdbook + run: | + mkdir mdbook + curl -Lf https://github.com/rust-lang/mdBook/releases/download/v0.4.3/mdbook-v0.4.3-x86_64-unknown-linux-gnu.tar.gz | tar -xz --directory=./mdbook + echo ::add-path::`pwd`/mdbook + - name: Deploy docs + run: | + cd src/doc/contrib + mdbook build + git worktree add gh-pages gh-pages + git config user.name "Deploy from CI" + git config user.email "" + cd gh-pages + # Delete the ref to avoid keeping history. + git update-ref -d refs/heads/gh-pages + rm -rf contrib + mv ../book contrib + git add contrib + git commit -m "Deploy $GITHUB_SHA to gh-pages" + git push --force diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md deleted file mode 100644 index f0acf00ad7e..00000000000 --- a/ARCHITECTURE.md +++ /dev/null @@ -1,150 +0,0 @@ -# Cargo Architecture - -This document gives a high level overview of Cargo internals. You may -find it useful if you want to contribute to Cargo or if you are -interested in the inner workings of Cargo. - -The purpose of Cargo is to formalize a canonical Rust workflow, by automating -the standard tasks associated with distributing software. Cargo simplifies -structuring a new project, adding dependencies, writing and running unit tests, -and more. - - -## Subcommands - -Cargo is a single binary composed of a set of [`clap`] subcommands. All subcommands live in -`src/bin/cargo/commands` directory. `src/bin/cargo/main.rs` is the entry point. - -Each subcommand, such as [`src/bin/cargo/commands/build.rs`], has its own API -interface, similarly to Git's, parsing command line options, reading the -configuration files, discovering the Cargo project in the current directory and -delegating the actual implementation to one -of the functions in [`src/cargo/ops/mod.rs`]. This short file is a good -place to find out about most of the things that Cargo can do. -Subcommands are designed to pipe to one another, and custom subcommands make -Cargo easy to extend and attach tools to. - -[`clap`]: https://clap.rs/ -[`src/bin/cargo/commands/build.rs`]: src/bin/cargo/commands/build.rs -[`src/cargo/ops/mod.rs`]: src/cargo/ops/mod.rs - - -## Important Data Structures - -There are some important data structures which are used throughout -Cargo. - -[`Config`] is available almost everywhere and holds "global" -information, such as `CARGO_HOME` or configuration from -`.cargo/config` files. The [`shell`] method of [`Config`] is the entry -point for printing status messages and other info to the console. - -[`Workspace`] is the description of the workspace for the current -working directory. Each workspace contains at least one -[`Package`]. Each package corresponds to a single `Cargo.toml`, and may -define several [`Target`]s, such as the library, binaries, integration -test or examples. Targets are crates (each target defines a crate -root, like `src/lib.rs` or `examples/foo.rs`) and are what is actually -compiled by `rustc`. - -A typical package defines the single library target and several -auxiliary ones. Packages are a unit of dependency in Cargo, and when -package `foo` depends on package `bar`, that means that each target -from `foo` needs the library target from `bar`. - -[`PackageId`] is the unique identifier of a (possibly remote) -package. It consist of three components: name, version and source -id. Source is the place where the source code for package comes -from. Typical sources are crates.io, a git repository or a folder on -the local hard drive. - -[`Resolve`] is the representation of a directed acyclic graph of package -dependencies, which uses [`PackageId`]s for nodes. This is the data -structure that is saved to the lock file. If there is no lock file, -Cargo constructs a resolve by finding a graph of packages which -matches declared dependency specification according to semver. - -[`Config`]: https://docs.rs/cargo/latest/cargo/util/config/struct.Config.html -[`shell`]: https://docs.rs/cargo/latest/cargo/util/config/struct.Config.html#method.shell -[`Workspace`]: https://docs.rs/cargo/latest/cargo/core/struct.Workspace.html -[`Package`]: https://docs.rs/cargo/latest/cargo/core/package/struct.Package.html -[`Target`]: https://docs.rs/cargo/latest/cargo/core/manifest/struct.Target.html -[`PackageId`]: https://docs.rs/cargo/latest/cargo/core/package_id/struct.PackageId.html -[`Resolve`]: https://docs.rs/cargo/latest/cargo/core/struct.Resolve.html - - -## Persistence - -Cargo is a non-daemon command line application, which means that all -the information used by Cargo must be persisted on the hard drive. The -main sources of information are `Cargo.toml` and `Cargo.lock` files, -`.cargo/config` configuration files and the globally shared registry -of packages downloaded from crates.io, usually located at -`~/.cargo/registry`. See [`src/cargo/sources/registry`] for the specifics of -the registry storage format. - -[`src/cargo/sources/registry`]: src/cargo/sources/registry - - -## Concurrency - -Cargo is mostly single threaded. The only concurrency inside a single -instance of Cargo happens during compilation, when several instances -of `rustc` are invoked in parallel to build independent -targets. However there can be several different instances of Cargo -process running concurrently on the system. Cargo guarantees that this -is always safe by using file locks when accessing potentially shared -data like the registry or the target directory. - - -## Tests - -Cargo has an impressive test suite located in the `tests` folder. Most -of the test are integration: a project structure with `Cargo.toml` and -rust source code is created in a temporary directory, `cargo` binary -is invoked via `std::process::Command` and then stdout and stderr are -verified against the expected output. To simplify testing, several -macros of the form `[MACRO]` are used in the expected output. For -example, `[..]` matches any string. - -To see stdout and stderr streams of the subordinate process, add `.stream()` -call to the built-up `Execs`: - -```rust -// Before -p.cargo("run").run(); - -// After -p.cargo("run").stream().run(); -``` - -Alternatively to build and run a custom version of cargo simply run `cargo build` -and execute `target/debug/cargo`. Note that `+nightly`/`+stable` (and variants), -being [rustup] features, won't work when executing the locally -built cargo binary directly, you have to instead build with `cargo +nightly build` -and run with `rustup run` (e.g `rustup run nightly -/target/debug/cargo ..`) (or set the `RUSTC` env var to point -to nightly rustc). - -[rustup]: https://rustup.rs/ - - -## Logging - -Cargo uses [`env_logger`], so you can set -`CARGO_LOG` environment variable to get the logs. This is useful both for diagnosing -bugs in stable Cargo and for local development. Cargo also has internal hierarchical -profiling infrastructure, which is activated via `CARGO_PROFILE` variable - -``` -# Outputs all logs with levels debug and higher -$ CARGO_LOG=debug cargo generate-lockfile - -# Don't forget that you can filter by module as well -$ CARGO_LOG=cargo::core::resolver=trace cargo generate-lockfile - -# Output first three levels of profiling info -$ CARGO_PROFILE=3 cargo generate-lockfile -``` - -[`env_logger`]: https://docs.rs/env_logger/*/env_logger/ diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 7c416fc1b6c..03993b9ff08 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,232 +1,5 @@ # Contributing to Cargo -Thank you for your interest in contributing to Cargo! Good places to -start are this document, [ARCHITECTURE.md](ARCHITECTURE.md), which -describes the high-level structure of Cargo and [E-easy] bugs on the -issue tracker. +Contributing documentation has moved to the **[Cargo Contributor Guide]**. -If you have a general question about Cargo or it's internals, feel free to ask -on [Zulip]. - -## Code of Conduct - -All contributors are expected to follow our [Code of Conduct]. - -## Bug reports - -We can't fix what we don't know about, so please report problems liberally. This -includes problems with understanding the documentation, unhelpful error messages -and unexpected behavior. - -**If you think that you have identified an issue with Cargo that might compromise -its users' security, please do not open a public issue on GitHub. Instead, -we ask you to refer to Rust's [security policy].** - -Opening an issue is as easy as following [this link][new-issues] and filling out -the fields. Here's a template that you can use to file an issue, though it's not -necessary to use it exactly: - - - - I tried this: - - I expected to see this happen: - - Instead, this happened: - - I'm using - -All three components are important: what you did, what you expected, what -happened instead. Please use https://gist.github.com/ if your examples run long. - - -## Feature requests - -Cargo follows the general Rust model of evolution. All major features go through -an RFC process. Therefore, before opening a feature request issue create a -Pre-RFC thread on the [internals][irlo] forum to get preliminary feedback. -Implementing a feature as a [custom subcommand][subcommands] is encouraged as it -helps demonstrate the demand for the functionality and is a great way to deliver -a working solution faster as it can iterate outside of cargo's release cadence. - -## Working on issues - -If you're looking for somewhere to start, check out the [E-easy][E-Easy] and -[E-mentor][E-mentor] tags. - -Feel free to ask for guidelines on how to tackle a problem on [Zulip] or open a -[new issue][new-issues]. This is especially important if you want to add new -features to Cargo or make large changes to the already existing code-base. -Cargo's core developers will do their best to provide help. - -If you start working on an already-filed issue, post a comment on this issue to -let people know that somebody is working it. Feel free to ask for comments if -you are unsure about the solution you would like to submit. - -While Cargo does make use of some Rust-features available only through the -`nightly` toolchain, it must compile on stable Rust. Code added to Cargo -is encouraged to make use of the latest stable features of the language and -`stdlib`. - -We use the "fork and pull" model [described here][development-models], where -contributors push changes to their personal fork and create pull requests to -bring those changes into the source repository. This process is partly -automated: Pull requests are made against Cargo's master-branch, tested and -reviewed. Once a change is approved to be merged, a friendly bot merges the -changes into an internal branch, runs the full test-suite on that branch -and only then merges into master. This ensures that Cargo's master branch -passes the test-suite at all times. - -Your basic steps to get going: - -* Fork Cargo and create a branch from master for the issue you are working on. -* Please adhere to the code style that you see around the location you are -working on. -* [Commit as you go][githelp]. -* Include tests that cover all non-trivial code. The existing tests -in `test/` provide templates on how to test Cargo's behavior in a -sandbox-environment. The internal module `crates/cargo-test-support` provides a vast amount -of helpers to minimize boilerplate. See [`crates/cargo-test-support/src/lib.rs`] for an -introduction to writing tests. -* Make sure `cargo test` passes. See [Running tests](#running-tests) below -for details on running tests. -* All code changes are expected to comply with the formatting suggested by `rustfmt`. -You can use `rustup component add rustfmt` to install `rustfmt` and use -`cargo fmt` to automatically format your code. -* Push your commits to GitHub and create a pull request against Cargo's -`master` branch. - -## Running tests - -Most of the tests are found in the `testsuite` integration test. This can be -run with a simple `cargo test`. - -Some tests only run on the nightly toolchain, and will be ignored on other -channels. It is recommended that you run tests with both nightly and stable to -ensure everything is working as expected. - -Some tests exercise cross compiling to a different target. This will require -you to install the appropriate target. This typically is the 32-bit target of -your host platform. For example, if your host is a 64-bit -`x86_64-unknown-linux-gnu`, then you should install the 32-bit target with -`rustup target add i686-unknown-linux-gnu`. If you don't have the alternate -target installed, there should be an error message telling you what to do. You -may also need to install additional tools for the target. For example, on Ubuntu -you should install the `gcc-multilib` package. - -If you can't install an alternate target, you can set the -`CFG_DISABLE_CROSS_TESTS=1` environment variable to disable these tests. The -Windows cross tests only support the MSVC toolchain. - -Some of the nightly tests require the `rustc-dev` and `llvm-tools-preview` -rustup components installed. These components include the compiler as a -library. This may already be installed with your nightly toolchain, but if it -isn't, run `rustup component add rustc-dev llvm-tools-preview ---toolchain=nightly`. - -There are several other packages in the repo for running specialized tests, -and you will need to run these tests separately by changing into its directory -and running `cargo test`: - -* [`crates/resolver-tests`] – This package runs tests on the dependency resolver. -* [`crates/cargo-platform`] – This is a standalone `cfg()` expression parser. - -The `build-std` tests are disabled by default, but you can run them by setting -the `CARGO_RUN_BUILD_STD_TESTS=1` environment variable and running `cargo test ---test build-std`. This requires the nightly channel, and also requires the -`rust-src` component installed with `rustup component add rust-src ---toolchain=nightly`. - -[`crates/resolver-tests`]: crates/resolver-tests -[`crates/cargo-platform`]: crates/cargo-platform - -## Pull requests - -After the pull request is made, a friendly bot will automatically assign a -reviewer; the review-process will make sure that the proposed changes are -sound. Please give the assigned reviewer sufficient time, especially during -weekends. If you don't get a reply, you may poke the core developers on [Zulip]. - -A merge of Cargo's master-branch and your changes is immediately queued -to be tested after the pull request is made. In case unforeseen -problems are discovered during this step (e.g., a failure on a platform you -originally did not develop on), you may ask for guidance. Push additional -commits to your branch to tackle these problems. - -The reviewer might point out changes deemed necessary. Please add them as -extra commits; this ensures that the reviewer can see what has changed since -the code was previously reviewed. Large or tricky changes may require several -passes of review and changes. - -Once the reviewer approves your pull request, a friendly bot picks it up -and [merges][mergequeue] it into Cargo's `master` branch. - -## Contributing to the documentation - -See the [documentation README](src/doc/README.md) for details on building the -documentation. - -## Issue Triage - -Sometimes an issue will stay open, even though the bug has been fixed. And -sometimes, the original bug may go stale because something has changed in the -meantime. - -It can be helpful to go through older bug reports and make sure that they are -still valid. Load up an older issue, double check that it's still true, and -leave a comment letting us know if it is or is not. The [least recently -updated sort][lru] is good for finding issues like this. - -Contributors with sufficient permissions on the Rust-repository can help by -adding labels to triage issues: - -* Yellow, **A**-prefixed labels state which **area** of the project an issue - relates to. - -* Magenta, **B**-prefixed labels identify bugs which are **blockers**. - -* Light purple, **C**-prefixed labels represent the **category** of an issue. - In particular, **C-feature-request** marks *proposals* for new features. If - an issue is **C-feature-request**, but is not **Feature accepted** or **I-nominated**, - then it was not thoroughly discussed, and might need some additional design - or perhaps should be implemented as an external subcommand first. Ping - @rust-lang/cargo if you want to send a PR for such issue. - -* Dark purple, **Command**-prefixed labels mean the issue has to do with a - specific cargo command. - -* Green, **E**-prefixed labels explain the level of **experience** or - **effort** necessary to fix the issue. [**E-mentor**][E-mentor] issues also - have some instructions on how to get started. - -* Red, **I**-prefixed labels indicate the **importance** of the issue. The - **[I-nominated][]** label indicates that an issue has been nominated for - prioritizing at the next triage meeting. - -* Purple gray, **O**-prefixed labels are the **operating system** or platform - that this issue is specific to. - -* Orange, **P**-prefixed labels indicate a bug's **priority**. These labels - are only assigned during triage meetings and replace the **[I-nominated][]** - label. - -* The light orange **relnotes** label marks issues that should be documented in - the release notes of the next release. - -* Dark blue, **Z**-prefixed labels are for unstable, nightly features. - -[githelp]: https://dont-be-afraid-to-commit.readthedocs.io/en/latest/git/commandlinegit.html -[development-models]: https://help.github.com/articles/about-collaborative-development-models/ -[gist]: https://gist.github.com/ -[new-issues]: https://github.com/rust-lang/cargo/issues/new -[mergequeue]: https://buildbot2.rust-lang.org/homu/queue/cargo -[security policy]: https://www.rust-lang.org/security.html -[lru]: https://github.com/rust-lang/cargo/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-asc -[E-easy]: https://github.com/rust-lang/cargo/labels/E-easy -[E-mentor]: https://github.com/rust-lang/cargo/labels/E-mentor -[I-nominated]: https://github.com/rust-lang/cargo/labels/I-nominated -[Code of Conduct]: https://www.rust-lang.org/conduct.html -[Zulip]: https://rust-lang.zulipchat.com/#narrow/stream/246057-t-cargo -[`crates/cargo-test-support/src/lib.rs`]: crates/cargo-test-support/src/lib.rs -[irlo]: https://internals.rust-lang.org/ -[subcommands]: https://doc.rust-lang.org/cargo/reference/external-tools.html#custom-subcommands +[Cargo Contributor Guide]: https://rust-lang.github.io/cargo/contrib/ diff --git a/README.md b/README.md index 535a975bfd9..6eed7578eb7 100644 --- a/README.md +++ b/README.md @@ -66,8 +66,10 @@ Please report all issues on the GitHub [issue tracker][issues]. ## Contributing -See [CONTRIBUTING.md](CONTRIBUTING.md). You may also find the architecture -documentation useful ([ARCHITECTURE.md](ARCHITECTURE.md)). +See the **[Cargo Contributor Guide]** for a complete introduction +to contributing to Cargo. + +[Cargo Contributor Guide]: https://rust-lang.github.io/cargo/contrib/ ## License diff --git a/crates/cargo-test-support/src/lib.rs b/crates/cargo-test-support/src/lib.rs index 57da10dd54b..a566561c385 100644 --- a/crates/cargo-test-support/src/lib.rs +++ b/crates/cargo-test-support/src/lib.rs @@ -1,109 +1,6 @@ -/* -# Introduction to `support`. - -Cargo has a wide variety of integration tests that execute the `cargo` binary -and verify its behavior. The `support` module contains many helpers to make -this process easy. - -The general form of a test involves creating a "project", running cargo, and -checking the result. Projects are created with the `ProjectBuilder` where you -specify some files to create. The general form looks like this: - -``` -let p = project() - .file("src/main.rs", r#"fn main() { println!("hi!"); }"#) - .build(); -``` - -If you do not specify a `Cargo.toml` manifest using `file()`, one is -automatically created with a project name of `foo` using `basic_manifest()`. - -To run cargo, call the `cargo` method and make assertions on the execution: - -``` -p.cargo("run --bin foo") - .with_stderr( - "\ -[COMPILING] foo [..] -[FINISHED] [..] -[RUNNING] `target/debug/foo` -", - ) - .with_stdout("hi!") - .run(); -``` - -The project creates a mini sandbox under the "cargo integration test" -directory with each test getting a separate directory such as -`/path/to/cargo/target/cit/t123/`. Each project appears as a separate -directory. There is also an empty `home` directory created that will be used -as a home directory instead of your normal home directory. - -See `support::lines_match` for an explanation of the string pattern matching. - -Browse the `pub` functions in the `support` module for a variety of other -helpful utilities. - -## Testing Nightly Features - -If you are testing a Cargo feature that only works on "nightly" cargo, then -you need to call `masquerade_as_nightly_cargo` on the process builder like -this: - -``` -p.cargo("build").masquerade_as_nightly_cargo() -``` - -If you are testing a feature that only works on *nightly rustc* (such as -benchmarks), then you should exit the test if it is not running with nightly -rust, like this: - -``` -if !is_nightly() { - // Add a comment here explaining why this is necessary. - return; -} -``` - -## Platform-specific Notes - -When checking output, use `/` for paths even on Windows: the actual output -of `\` on Windows will be replaced with `/`. - -Be careful when executing binaries on Windows. You should not rename, delete, -or overwrite a binary immediately after running it. Under some conditions -Windows will fail with errors like "directory not empty" or "failed to remove" -or "access is denied". - -## Specifying Dependencies - -You should not write any tests that use the network such as contacting -crates.io. Typically, simple path dependencies are the easiest way to add a -dependency. Example: - -``` -let p = project() - .file("Cargo.toml", r#" - [package] - name = "foo" - version = "1.0.0" - - [dependencies] - bar = {path = "bar"} - "#) - .file("src/lib.rs", "extern crate bar;") - .file("bar/Cargo.toml", &basic_manifest("bar", "1.0.0")) - .file("bar/src/lib.rs", "") - .build(); -``` - -If you need to test with registry dependencies, see -`support::registry::Package` for creating packages you can depend on. - -If you need to test git dependencies, see `support::git` to create a git -dependency. - -*/ +//! # Cargo test support. +//! +//! See https://rust-lang.github.io/cargo/contrib/ for a guide on writing tests. #![allow(clippy::needless_doctest_main)] // according to @ehuss this lint is fussy #![allow(clippy::inefficient_to_string)] // this causes suggestions that result in `(*s).to_string()` diff --git a/src/doc/contrib/README.md b/src/doc/contrib/README.md new file mode 100644 index 00000000000..57756211b25 --- /dev/null +++ b/src/doc/contrib/README.md @@ -0,0 +1,12 @@ +# Cargo Contributor Guide + +This is the source of the Cargo Contributor Guide, published at +. It is written in Markdown, using +the [mdbook] tool to convert to HTML. If you are editing these pages, the best +option to view the results is to run `mdbook serve`, which will start a web +server on localhost that you can visit to view the book, and it will +automatically reload each time you edit a page. + +This is published via GitHub Actions to GitHub Pages. + +[mdbook]: https://rust-lang.github.io/mdBook/ diff --git a/src/doc/contrib/book.toml b/src/doc/contrib/book.toml new file mode 100644 index 00000000000..28c143da6b5 --- /dev/null +++ b/src/doc/contrib/book.toml @@ -0,0 +1,3 @@ +[book] +title = "Cargo Contributor Guide" +authors = ["Eric Huss"] diff --git a/src/doc/contrib/src/SUMMARY.md b/src/doc/contrib/src/SUMMARY.md new file mode 100644 index 00000000000..4a48c03070e --- /dev/null +++ b/src/doc/contrib/src/SUMMARY.md @@ -0,0 +1,20 @@ +# Summary + +- [Introduction](./index.md) +- [Issue Tracker](./issues.md) +- [Process](./process/index.md) + - [Working on Cargo](./process/working-on-cargo.md) + - [Release process](./process/release.md) + - [Unstable features](./process/unstable.md) +- [Architecture](./architecture/index.md) + - [Codebase Overview](./architecture/codebase.md) + - [SubCommands](./architecture/subcommands.md) + - [Console Output](./architecture/console.md) + - [Packages and Resolution](./architecture/packages.md) + - [Compilation](./architecture/compilation.md) + - [Files](./architecture/files.md) +- [Tests](./tests/index.md) + - [Running Tests](./tests/running.md) + - [Writing Tests](./tests/writing.md) + - [Profiling](./tests/profiling.md) +- [Design Principles](./design.md) diff --git a/src/doc/contrib/src/architecture/codebase.md b/src/doc/contrib/src/architecture/codebase.md new file mode 100644 index 00000000000..f81327402e7 --- /dev/null +++ b/src/doc/contrib/src/architecture/codebase.md @@ -0,0 +1,72 @@ +# Codebase Overview + +This is a very high-level overview of the Cargo codebase. + +* [`src/bin/cargo`](https://github.com/rust-lang/cargo/tree/master/src/bin/cargo) + — Cargo is split in a library and a binary. This is the binary side that + handles argument parsing, and then calls into the library to perform the + appropriate subcommand. Each Cargo subcommand is a separate module here. See + [SubCommands](subcommands.md). + +* [`src/cargo/ops`](https://github.com/rust-lang/cargo/tree/master/src/cargo/ops) + — Every major operation is implemented here. This is where the binary CLI + usually calls into to perform the appropriate action. + + * [`src/cargo/ops/cargo_compile.rs`](https://github.com/rust-lang/cargo/blob/master/src/cargo/ops/cargo_compile.rs) + — This is the entry point for all the compilation commands. This is a + good place to start if you want to follow how compilation starts and + flows to completion. + +* [`src/cargo/core/resolver`](https://github.com/rust-lang/cargo/tree/master/src/cargo/core/resolver) + — This is the dependency and feature resolvers. + +* [`src/cargo/core/compiler`](https://github.com/rust-lang/cargo/tree/master/src/cargo/core/compiler) + — This is the code responsible for running `rustc` and `rustdoc`. + + * [`src/cargo/core/compiler/build_context/mod.rs`](https://github.com/rust-lang/cargo/blob/master/src/cargo/core/compiler/build_context/mod.rs) + — The `BuildContext` is the result of the "front end" of the build + process. This contains the graph of work to perform and any settings + necessary for `rustc`. After this is built, the next stage of building + is handled in `Context`. + + * [`src/cargo/core/compiler/context`](https://github.com/rust-lang/cargo/blob/master/src/cargo/core/compiler/context/mod.rs) + — The `Context` is the mutable state used during the build process. This + is the core of the build process, and everything is coordinated through + this. + + * [`src/cargo/core/compiler/fingerprint.rs`](https://github.com/rust-lang/cargo/blob/master/src/cargo/core/compiler/fingerprint.rs) + — The `fingerprint` module contains all the code that handles detecting + if a crate needs to be recompiled. + +* [`src/cargo/core/source`](https://github.com/rust-lang/cargo/tree/master/src/cargo/core/source) + — The `Source` trait is an abstraction over different sources of packages. + Sources are uniquely identified by a `SourceId`. Sources are implemented in + the + [`src/cargo/sources`](https://github.com/rust-lang/cargo/tree/master/src/cargo/sources) + directory. + +* [`src/cargo/util`](https://github.com/rust-lang/cargo/tree/master/src/cargo/util) + — This directory contains generally-useful utility modules. + +* [`src/cargo/util/config`](https://github.com/rust-lang/cargo/tree/master/src/cargo/util/config) + — This directory contains the config parser. It makes heavy use of + [serde](https://serde.rs/) to merge and translate config values. The + `Config` is usually accessed from the + [`Workspace`](https://github.com/rust-lang/cargo/blob/master/src/cargo/core/workspace.rs), + though references to it are scattered around for more convenient access. + +* [`src/cargo/util/toml`](https://github.com/rust-lang/cargo/tree/master/src/cargo/util/toml) + — This directory contains the code for parsing `Cargo.toml` files. + + * [`src/cargo/ops/lockfile.rs`](https://github.com/rust-lang/cargo/blob/master/src/cargo/ops/lockfile.rs) + — This is where `Cargo.lock` files are loaded and saved. + +* [`src/doc`](https://github.com/rust-lang/cargo/tree/master/src/doc) + — This directory contains Cargo's documentation and man pages. + +* [`src/etc`](https://github.com/rust-lang/cargo/tree/master/src/etc) + — These are files that get distributed in the `etc` directory in the Rust release. + The man pages are auto-generated by a script in the `src/doc` directory. + +* [`crates`](https://github.com/rust-lang/cargo/tree/master/crates) + — A collection of independent crates used by Cargo. diff --git a/src/doc/contrib/src/architecture/compilation.md b/src/doc/contrib/src/architecture/compilation.md new file mode 100644 index 00000000000..c88d6567a29 --- /dev/null +++ b/src/doc/contrib/src/architecture/compilation.md @@ -0,0 +1,39 @@ +# Compilation + +The [`Unit`] is the primary data structure representing a single execution of +the compiler. It (mostly) contains all the information needed to determine +which flags to pass to the compiler. + +The entry to the compilation process is located in the [`cargo_compile`] +module. The compilation can be conceptually broken into these steps: + +1. Perform dependency resolution (see [the resolution chapter]). +2. Generate the root `Unit`s, the things the user requested to compile on the + command-line. This is done in [`generate_targets`]. +3. Starting from the root `Unit`s, generate the [`UnitGraph`] by walking the + dependency graph from the resolver. The `UnitGraph` contains all of the + `Unit` structs, and information about the dependency relationships between + units. This is done in the [`unit_dependencies`] module. +4. Construct the [`BuildContext`] with all of the information collected so + far. This is the end of the "front end" of compilation. +5. Create a [`Context`], a large, mutable data structure that coordinates the + compilation process. +6. The [`Context`] will create a [`JobQueue`], a data structure that tracks + which units need to be built. +7. [`drain_the_queue`] does the compilation process. This is the only point in + Cargo that currently uses threads. +8. The result of the compilation is stored in the [`Compilation`] struct. This + can be used for various things, such as running tests after the compilation + has finished. + +[`cargo_compile`]: https://github.com/rust-lang/cargo/blob/master/src/cargo/ops/cargo_compile.rs +[`generate_targets`]: https://github.com/rust-lang/cargo/blob/e4b65bdc80f2a293447f2f6a808fa7c84bf9a357/src/cargo/ops/cargo_compile.rs#L725-L739 +[`UnitGraph`]: https://github.com/rust-lang/cargo/blob/master/src/cargo/core/compiler/unit_graph.rs +[the resolution chapter]: packages.md +[`Unit`]: https://github.com/rust-lang/cargo/blob/master/src/cargo/core/compiler/unit.rs +[`unit_dependencies`]: https://github.com/rust-lang/cargo/blob/master/src/cargo/core/compiler/unit_dependencies.rs +[`BuildContext`]: https://github.com/rust-lang/cargo/blob/master/src/cargo/core/compiler/build_context/mod.rs +[`Context`]: https://github.com/rust-lang/cargo/blob/master/src/cargo/core/compiler/context/mod.rs +[`JobQueue`]: https://github.com/rust-lang/cargo/blob/master/src/cargo/core/compiler/job_queue.rs +[`drain_the_queue`]: https://github.com/rust-lang/cargo/blob/e4b65bdc80f2a293447f2f6a808fa7c84bf9a357/src/cargo/core/compiler/job_queue.rs#L623-L634 +[`Compilation`]: https://github.com/rust-lang/cargo/blob/master/src/cargo/core/compiler/compilation.rs diff --git a/src/doc/contrib/src/architecture/console.md b/src/doc/contrib/src/architecture/console.md new file mode 100644 index 00000000000..2c5412b8c46 --- /dev/null +++ b/src/doc/contrib/src/architecture/console.md @@ -0,0 +1,82 @@ +# Console Output + +All of Cargo's output should go through the [`Shell`] struct. You can normally +obtain the `Shell` instance from the [`Config`] struct. Do **not** use the std +`println!` macros. + +Most of Cargo's output goes to stderr. When running in JSON mode, the output +goes to stdout. + +It is important to properly handle errors when writing to the console. +Informational commands, like `cargo list`, should ignore any errors writing +the output. There are some [`drop_print`] macros that are intended to make +this easier. + +Messages written during compilation should handle errors, and abort the build +if they are unable to be displayed. This is generally automatically handled in +the [`JobQueue`] as it processes each message. + +[`Shell`]: https://github.com/rust-lang/cargo/blob/master/src/cargo/core/shell.rs +[`Config`]: https://github.com/rust-lang/cargo/blob/master/src/cargo/util/config/mod.rs +[`drop_print`]: https://github.com/rust-lang/cargo/blob/e4b65bdc80f2a293447f2f6a808fa7c84bf9a357/src/cargo/util/config/mod.rs#L1820-L1848 +[`JobQueue`]: https://github.com/rust-lang/cargo/blob/master/src/cargo/core/compiler/job_queue.rs + +## Errors + +Cargo uses [`anyhow`] for managing errors. This makes it convenient to "chain" +errors together, so that Cargo can report how an error originated, and what it +was trying to do at the time. + +Error helpers are implemented in the [`errors`] module. Use the +`InternalError` error type for errors that are not expected to happen. This +will print a message to the user to file a bug report. + +The binary side of Cargo uses the `CliError` struct to wrap the process exit +code. Usually Cargo exits with 101 for an error, but some commands like `cargo +test` will exit with different codes. + +[`errors`]: https://github.com/rust-lang/cargo/blob/master/src/cargo/util/errors.rs + +## Style + +Some guidelines for Cargo's output: + +* Keep the normal output brief. Cargo is already fairly noisy, so try to keep + the output as brief and clean as possible. +* Good error messages are very important! Try to keep them brief and to the + point, but good enough that a beginner can understand what is wrong and can + figure out how to fix. It is a difficult balance to hit! Err on the side of + providing extra information. +* When using any low-level routines, such as `std::fs`, *always* add error + context about what it is doing. For example, reading from a file should + include context about which file is being read if there is an error. +* Cargo's error style is usually a phrase, starting with a lowercase letter. + If there is a longer error message that needs multiple sentences, go ahead + and use multiple sentences. This should probably be improved sometime in the + future to be more structured. + +## Debug logging + +Cargo uses the [`env_logger`] crate to display debug log messages. The +`CARGO_LOG` environment variable can be set to enable debug logging, with a +value such as `trace`, `debug`, or `warn`. It also supports filtering for +specific modules. Feel free to use the standard [`log`] macros to help with +diagnosing problems. + +```sh +# Outputs all logs with levels debug and higher +CARGO_LOG=debug cargo generate-lockfile + +# Don't forget that you can filter by module as well +CARGO_LOG=cargo::core::resolver=trace cargo generate-lockfile + +# This will print lots of info about the download process. `trace` prints even more. +CARGO_HTTP_DEBUG=true CARGO_LOG=cargo::ops::registry=debug cargo fetch + +# This is an important command for diagnosing fingerprint issues. +CARGO_LOG=cargo::core::compiler::fingerprint=trace cargo build +``` + +[`env_logger`]: https://docs.rs/env_logger +[`log`]: https://docs.rs/log +[`anyhow`]: https://docs.rs/anyhow diff --git a/src/doc/contrib/src/architecture/files.md b/src/doc/contrib/src/architecture/files.md new file mode 100644 index 00000000000..2e6e02b0790 --- /dev/null +++ b/src/doc/contrib/src/architecture/files.md @@ -0,0 +1,67 @@ +# Files + +This chapter gives some pointers on where to start looking at Cargo's on-disk +data file structures. + +* [`Layout`] is the abstraction for the `target` directory. It handles locking + the target directory, and providing paths to the parts inside. There is a + separate `Layout` for each "target". +* [`Resolve`] contains the contents of the `Cargo.lock` file. See the [`encode`] + module for the different `Cargo.lock` formats. +* [`TomlManifest`] contains the contents of the `Cargo.toml` file. It is translated + to a [`Manifest`] object for some simplification, and the `Manifest` is stored + in a [`Package`]. +* The [`fingerprint`] module deals with the fingerprint information stored in + `target/debug/.fingerprint`. This tracks whether or not a crate needs to be + rebuilt. +* `cargo install` tracks its installed files with some metadata in + `$CARGO_HOME`. The metadata is managed in the + [`common_for_install_and_uninstall`] module. +* Git sources are cached in `$CARGO_HOME/git`. The code for this cache is in + the [`git`] source module. +* Registries are cached in `$CARGO_HOME/registry`. There are three parts, the + index, the compressed `.crate` files, and the extracted sources of those + crate files. + * Management of the registry cache can be found in the [`registry`] source + module. Note that this includes an on-disk cache as an optimization for + accessing the git repository. + * Saving of `.crate` files is handled by the [`RemoteRegistry`]. + * Extraction of `.crate` files is handled by the [`RegistrySource`]. + * There is a lock for the package cache. Code must be careful, because + this lock must be obtained manually. See + [`Config::acquire_package_cache_lock`]. + +[`Layout`]: https://github.com/rust-lang/cargo/blob/master/src/cargo/core/compiler/layout.rs +[`Resolve`]: https://github.com/rust-lang/cargo/blob/master/src/cargo/core/resolver/resolve.rs +[`encode`]: https://github.com/rust-lang/cargo/blob/master/src/cargo/core/resolver/encode.rs +[`TomlManifest`]: https://github.com/rust-lang/cargo/blob/master/src/cargo/util/toml/mod.rs +[`Manifest`]: https://github.com/rust-lang/cargo/blob/master/src/cargo/core/manifest.rs +[`Package`]: https://github.com/rust-lang/cargo/blob/master/src/cargo/core/package.rs +[`common_for_install_and_uninstall`]: https://github.com/rust-lang/cargo/blob/master/src/cargo/ops/common_for_install_and_uninstall.rs +[`git`]: https://github.com/rust-lang/cargo/tree/master/src/cargo/sources/git +[`registry`]: https://github.com/rust-lang/cargo/blob/master/src/cargo/sources/registry/mod.rs +[`RemoteRegistry`]: https://github.com/rust-lang/cargo/blob/master/src/cargo/sources/registry/remote.rs +[`RegistrySource`]: https://github.com/rust-lang/cargo/blob/master/src/cargo/sources/registry/mod.rs +[`Config::acquire_package_cache_lock`]: https://github.com/rust-lang/cargo/blob/e4b65bdc80f2a293447f2f6a808fa7c84bf9a357/src/cargo/util/config/mod.rs#L1261-L1266 + +## Filesystems + +Cargo tends to get run on a very wide array of file systems. Different file +systems can have a wide range of capabilities, and Cargo should strive to do +its best to handle them. Some examples of issues to deal with: + +* Not all file systems support locking. Cargo tries to detect if locking is + supported, and if not, will ignore lock errors. This isn't ideal, but it is + difficult to deal with. +* The [`fs::canonicalize`] function doesn't work on all file systems + (particularly some Windows file systems). If that function is used, there + should be a fallback if it fails. This function will also return `\\?\` + style paths on Windows, which can have some issues (such as some tools not + supporting them, or having issues with relative paths). +* Timestamps can be unreliable. The [`fingerprint`] module has a deeper + discussion of this. One example is that Docker cache layers will erase the + fractional part of the time stamp. +* Symlinks are not always supported, particularly on Windows. + +[`fingerprint`]: https://github.com/rust-lang/cargo/blob/master/src/cargo/core/compiler/fingerprint.rs +[`fs::canonicalize`]: https://doc.rust-lang.org/std/fs/fn.canonicalize.html diff --git a/src/doc/contrib/src/architecture/index.md b/src/doc/contrib/src/architecture/index.md new file mode 100644 index 00000000000..42e34a2c8e6 --- /dev/null +++ b/src/doc/contrib/src/architecture/index.md @@ -0,0 +1 @@ +# Architecture Overview diff --git a/src/doc/contrib/src/architecture/packages.md b/src/doc/contrib/src/architecture/packages.md new file mode 100644 index 00000000000..a91a8e4742b --- /dev/null +++ b/src/doc/contrib/src/architecture/packages.md @@ -0,0 +1,92 @@ +# Packages and Resolution + +## Workspaces + +The [`Workspace`] object is usually created very early by calling the +[`workspace`][ws-method] helper method. This discovers the root of the +workspace, and loads all the workspace members as a [`Package`] object. Each +package corresponds to a single `Cargo.toml` (which is deserialized into a +[`Manifest`]), and may define several [`Target`]s, such as the library, +binaries, integration test or examples. Targets are crates (each target +defines a crate root, like `src/lib.rs` or `examples/foo.rs`) and are what is +actually compiled by `rustc`. + +## Packages and Sources + +There are several data structures that are important to understand how +packages are found and loaded: + +* [`Package`] — A package, which is a `Cargo.toml` manifest and its associated + source files. + * [`PackageId`] — A unique identifier for a package. +* [`Source`] — An abstraction for something that can fetch packages (a remote + registry, a git repo, the local filesystem, etc.). Check out the [source + implementations] for all the details about registries, indexes, git + dependencies, etc. + * [`SourceId`] — A unique identifier for a source. +* [`SourceMap`] — Map of all available sources. +* [`PackageRegistry`] — This is the main interface for how the dependency + resolver finds packages. It contains the `SourceMap`, and handles things + like the `[patch]` table. The `Registry` trait provides a generic interface + to the `PackageRegistry`, but this is only used for providing an alternate + implementation of the `PackageRegistry` for testing. The dependency resolver + sends a query to the `PackageRegistry` to "get me all packages that match + this dependency declaration". +* [`Summary`] — A summary is a subset of a [`Manifest`], and is essentially + the information that can be found in a registry index. Queries against the + `PackageRegistry` yields a `Summary`. The resolver uses the summary + information to build the dependency graph. +* [`PackageSet`] — Contains all of the `Package` objects. This works with the + [`Downloads`] struct to coordinate downloading packages. It has a reference + to the `SourceMap` to get the `Source` objects which tell the `Downloads` + struct which URLs to fetch. + +All of these come together in the [`ops::resolve`] module. This module +contains the primary functions for performing resolution (described below). It +also handles downloading of packages. It is essentially where all of the data +structures above come together. + +## Resolver + +[`Resolve`] is the representation of a directed graph of package dependencies, +which uses [`PackageId`]s for nodes. This is the data structure that is saved +to the `Cargo.lock` file. If there is no lock file, Cargo constructs a resolve +by finding a graph of packages which matches declared dependency specification +according to SemVer. + +[`ops::resolve`] is the front-end for creating a `Resolve`. It handles loading +the `Cargo.lock` file, checking if it needs updating, etc. + +Resolution is currently performed twice. It is performed once with all +features enabled. This is the resolve that gets saved to `Cargo.lock`. It then +runs again with only the specific features the user selected on the +command-line. Ideally this second run will get removed in the future when +transitioning to the new feature resolver. + +### Feature resolver + +A new feature-specific resolver was added in 2020 which adds more +sophisticated feature resolution. It is located in the [`resolver::features`] +module. The original dependency resolver still performs feature unification, +as it can help reduce the dependencies it has to consider during resolution +(rather than assuming every optional dependency of every package is enabled). +Checking if a feature is enabled must go through the new feature resolver. + + +[`Workspace`]: https://github.com/rust-lang/cargo/blob/master/src/cargo/core/workspace.rs +[ws-method]: https://github.com/rust-lang/cargo/blob/e4b65bdc80f2a293447f2f6a808fa7c84bf9a357/src/cargo/util/command_prelude.rs#L298-L318 +[`Package`]: https://github.com/rust-lang/cargo/blob/master/src/cargo/core/package.rs +[`Target`]: https://github.com/rust-lang/cargo/blob/e4b65bdc80f2a293447f2f6a808fa7c84bf9a357/src/cargo/core/manifest.rs#L181-L206 +[`Manifest`]: https://github.com/rust-lang/cargo/blob/e4b65bdc80f2a293447f2f6a808fa7c84bf9a357/src/cargo/core/manifest.rs#L27-L51 +[`Source`]: https://github.com/rust-lang/cargo/blob/master/src/cargo/core/source/mod.rs +[`SourceId`]: https://github.com/rust-lang/cargo/blob/master/src/cargo/core/source/source_id.rs +[`SourceMap`]: https://github.com/rust-lang/cargo/blob/e4b65bdc80f2a293447f2f6a808fa7c84bf9a357/src/cargo/core/source/mod.rs#L245-L249 +[`PackageRegistry`]: https://github.com/rust-lang/cargo/blob/e4b65bdc80f2a293447f2f6a808fa7c84bf9a357/src/cargo/core/registry.rs#L36-L81 +[`ops::resolve`]: https://github.com/rust-lang/cargo/blob/master/src/cargo/ops/resolve.rs +[`resolver::features`]: https://github.com/rust-lang/cargo/blob/master/src/cargo/core/resolver/features.rs#L259 +[source implementations]: https://github.com/rust-lang/cargo/tree/master/src/cargo/sources +[`PackageId`]: https://github.com/rust-lang/cargo/blob/master/src/cargo/core/package_id.rs +[`Summary`]: https://github.com/rust-lang/cargo/blob/master/src/cargo/core/summary.rs +[`PackageSet`]: https://github.com/rust-lang/cargo/blob/e4b65bdc80f2a293447f2f6a808fa7c84bf9a357/src/cargo/core/package.rs#L283-L296 +[`Downloads`]: https://github.com/rust-lang/cargo/blob/e4b65bdc80f2a293447f2f6a808fa7c84bf9a357/src/cargo/core/package.rs#L298-L352 +[`Resolve`]: https://github.com/rust-lang/cargo/blob/master/src/cargo/core/resolver/resolve.rs diff --git a/src/doc/contrib/src/architecture/subcommands.md b/src/doc/contrib/src/architecture/subcommands.md new file mode 100644 index 00000000000..85b0af37158 --- /dev/null +++ b/src/doc/contrib/src/architecture/subcommands.md @@ -0,0 +1,25 @@ +# SubCommands + +Cargo is a single binary composed of a set of [`clap`] subcommands. All +subcommands live in [`src/bin/cargo/commands`] directory. +[`src/bin/cargo/main.rs`] is the entry point. + +Each subcommand, such as [`src/bin/cargo/commands/build.rs`], usually performs +the following: + +* Parse the CLI flags. See the [`command_prelude`] module for some helpers to make this easier. +* Load the config files. +* Discover and load the workspace. +* Calls the actual implementation of the subcommand which resides in [`src/cargo/ops`]. + +If the subcommand is not found in the built-in list, then Cargo will +automatically search for a subcommand named `cargo-{NAME}` in the users `PATH` +to execute the subcommand. + + +[`clap`]: https://clap.rs/ +[`src/bin/cargo/commands/build.rs`]: https://github.com/rust-lang/cargo/tree/master/src/bin/cargo/commands/build.rs +[`src/cargo/ops`]: https://github.com/rust-lang/cargo/tree/master/src/cargo/ops +[`src/bin/cargo/commands`]: https://github.com/rust-lang/cargo/tree/master/src/bin/cargo/commands +[`src/bin/cargo/main.rs`]: https://github.com/rust-lang/cargo/blob/master/src/bin/cargo/main.rs +[`command_prelude`]: https://github.com/rust-lang/cargo/blob/master/src/cargo/util/command_prelude.rs diff --git a/src/doc/contrib/src/design.md b/src/doc/contrib/src/design.md new file mode 100644 index 00000000000..d51d3eb2085 --- /dev/null +++ b/src/doc/contrib/src/design.md @@ -0,0 +1,101 @@ +# Design Principles + +The purpose of Cargo is to formalize a canonical Rust workflow, by automating +the standard tasks associated with distributing software. Cargo simplifies +structuring a new project, adding dependencies, writing and running unit +tests, and more. + +Cargo is not intended to be a general-purpose build tool. Ideally, it should +be easy to integrate it within another build tool, though admittedly that is +not as seamless as desired. + +## Stability and compatibility + +### Backwards compatibility + +Cargo strives to remain backwards compatible with projects created in previous +versions. The CLI interface also strives to remain backwards compatible, such +that the commands and options behave the same. That being said, changes in +behavior, and even outright breakage are sometimes done in limited situations. +The following outlines some situations where backwards-incompatible changes are +made: + +* Anything that addresses a security concern. +* Dropping support for older platforms and tooling. Cargo follows the Rust + [tiered platform support]. +* Changes to resolve possibly unsafe or unreliable behavior. + +None of these changes should be taken lightly, and should be avoided if +possible, or possibly with some transition period to alert the user of the +potential change. + +Behavior is sometimes changed in ways that have a high confidence that it +won't break existing workflows. Almost every change carries this risk, so it +is often a judgment call balancing the benefit of the change with the +perceived possibility of its negative consequences. + +At times, some changes fall in the gray area, where the current behavior is +undocumented, or not working as intended. These are more difficult judgment +calls. The general preference is to balance towards avoiding breaking existing +workflows. + +Support for older registry APIs and index formats may be dropped, if there is +high confidence that there aren't any active registries that may be affected. +This has never (to my knowledge) happened so far, and is unlikely to happen in +the future, but remains a possibility. + +In all of the above, a transition period may be employed if a change is known +to cause breakage. A warning can be issued to alert the user that something +will change, and provide them with an alternative to resolve the issue +(preferably in a way that is compatible across versions if possible). + +Cargo is only expected to work with the version of the related Rust tools +(`rustc`, `rustdoc`, etc.) that it is released with. As a matter of choice, +the latest nightly works with the most recent stable release, but that is +mostly to accommodate development of Cargo itself, and should not be expected +by users. + +### Forwards compatibility + +Additionally, Cargo strives a limited degree of *forwards compatibility*. +Changes should not egregiously prevent older versions from working. This is +mostly relevant for persistent data, such as on-disk files and the registry +interface and index. It also applies to a lesser degree to the registry API. + +Changes to `Cargo.lock` require a transition time, where the new format is not +automatically written when the lock file is updated. The transition time +should not be less than 6 months, though preferably longer. New projects may +use the new format in a shorter time frame. + +Changes to `Cargo.toml` can be made in any release. This is because the user +must manually modify the file, and opt-in to any new changes. Additionally, +Cargo will usually only issue a warning about new fields it doesn't +understand, but otherwise continue to function. + +Changes to cache files (such as artifacts in the `target` directory, or cached +data in Cargo's home directory) should not *prevent* older versions from +running, but they may cause older versions to recreate the cache, which may +result in a performance impact. + +Changes to the registry index should not prevent older versions from working. +Generally, older versions ignore new fields, so the format should be easily +extensible. Changes to the format or interpretation of existing fields should +be done very carefully to avoid preventing older versions of Cargo from +working. In some cases, this may mean that older versions of Cargo will not be +able to *select* a newly published crate, but it shouldn't prevent them from +working at all. This level of compatibility may not last forever, but the +exact time frame for such a change has not yet been decided. + +The registry API may be changed in such a way to prevent older versions of +Cargo from working. Generally, compatibility should be retained for as long as +possible, but the exact length of time is not specified. + +## Simplicity and layers + +Standard workflows should be easy and consistent. Each knob that is added has +a high cost, regardless if it is intended for a small audience. Layering and +defaults can help avoid the surface area that the user needs to be concerned +with. Try to avoid small functionalities that may have complex interactions +with one another. + +[tiered platform support]: https://doc.rust-lang.org/nightly/rustc/platform-support.html diff --git a/src/doc/contrib/src/index.md b/src/doc/contrib/src/index.md new file mode 100644 index 00000000000..6fb8718d3ba --- /dev/null +++ b/src/doc/contrib/src/index.md @@ -0,0 +1,29 @@ +# Introduction + +Thank you for your interest in contributing to [Cargo]! This guide provides an +overview of how to contribute to Cargo, how to dive into the code, and how the +testing infrastructure works. + +There are many ways to contribute, such as [helping other users], [filing +issues], [improving the documentation], [fixing bugs], and working on [small] +and [large features]. + +If you have a general question about Cargo or it's internals, feel free to ask +on [Zulip]. + +This guide assumes you have some familiarity with Rust, and how to use Cargo, +[rustup], and general development tools like [git]. + +Please also read the [Rust Code of Conduct]. + +[Cargo]: https://doc.rust-lang.org/cargo/ +[Zulip]: https://rust-lang.zulipchat.com/#narrow/stream/246057-t-cargo +[Rust Code of Conduct]: https://www.rust-lang.org/policies/code-of-conduct +[helping other users]: https://users.rust-lang.org/ +[filing issues]: issues.md +[rustup]: https://rust-lang.github.io/rustup/ +[git]: https://git-scm.com/ +[improving the documentation]: https://github.com/rust-lang/cargo/tree/master/src/doc +[fixing bugs]: process/index.md#working-on-small-bugs +[small]: process/index.md#working-on-small-features +[large features]: process/index.md#working-on-large-features diff --git a/src/doc/contrib/src/issues.md b/src/doc/contrib/src/issues.md new file mode 100644 index 00000000000..73f60643abc --- /dev/null +++ b/src/doc/contrib/src/issues.md @@ -0,0 +1,109 @@ +# Issue Tracker + +Cargo's issue tracker is located at +. This is the primary spot where +we track bugs and small feature requests. See [Process] for more about our +process for proposing changes. + +## Filing issues + +We can't fix what we don't know about, so please report problems liberally. +This includes problems with understanding the documentation, unhelpful error +messages, and unexpected behavior. + +**If you think that you have identified an issue with Cargo that might +compromise its users' security, please do not open a public issue on GitHub. +Instead, we ask you to refer to Rust's [security policy].** + +Opening an issue is as easy as following [this link][new-issues]. There are +several templates for different issue kinds, but if none of them fit your +issue, don't hesitate to modify one of the templates, or click the [Open a +blank issue] link. + +The Rust tools are spread across multiple repositories in the Rust +organization. It may not always be clear where to file an issue. No worries! +If you file in the wrong tracker, someone will either transfer it to the +correct one or ask you to move it. Some other repositories that may be +relevant are: + +* [`rust-lang/rust`] — Home for the [`rustc`] compiler and [`rustdoc`]. +* [`rust-lang/rustup`] — Home for the [`rustup`] toolchain installer. +* [`rust-lang/rustfmt`] — Home for the `rustfmt` tool, which also includes `cargo fmt`. +* [`rust-lang/rust-clippy`] — Home for the `clippy` tool, which also includes `cargo clippy`. +* [`rust-lang/crates.io`] — Home for the [crates.io] website. + +Issues with [`cargo fix`] can be tricky to know where they should be filed, +since the fixes are driven by `rustc`, processed by [`rustfix`], and the +front-interface is implemented in Cargo. Feel free to file in the Cargo issue +tracker, and it will get moved to one of the other issue trackers if +necessary. + +[Process]: process/index.md +[security policy]: https://www.rust-lang.org/security.html +[new-issues]: https://github.com/rust-lang/cargo/issues/new/choose +[Open a blank issue]: https://github.com/rust-lang/cargo/issues/new +[`rust-lang/rust`]: https://github.com/rust-lang/rust +[`rust-lang/rustup`]: https://github.com/rust-lang/rustup +[`rust-lang/rustfmt`]: https://github.com/rust-lang/rustfmt +[`rust-lang/rust-clippy`]: https://github.com/rust-lang/rust-clippy +[`rustc`]: https://doc.rust-lang.org/rustc/ +[`rustdoc`]: https://doc.rust-lang.org/rustdoc/ +[`rustup`]: https://rust-lang.github.io/rustup/ +[`rust-lang/crates.io`]: https://github.com/rust-lang/crates.io +[crates.io]: https://crates.io/ +[`rustfix`]: https://github.com/rust-lang/rustfix/ +[`cargo fix`]: https://doc.rust-lang.org/cargo/commands/cargo-fix.html + +## Issue labels + +[Issue labels] are very helpful to identify the types of issues and which +category they are related to. The Cargo team typically manages assigning +labels. The labels use a naming convention with short prefixes and colors to +indicate the kind of label: + +* Yellow, **A**-prefixed labels state which **area** of the project an issue + relates to. + +* Light purple, **C**-prefixed labels represent the **category** of an issue. + In particular, **[C-feature-request]** marks *proposals* for new features. If + an issue is **C-feature-request**, but is not **[Feature accepted]** or + **[I-nominated]**, then it was not thoroughly discussed, and might need some + additional design or perhaps should be implemented as an external subcommand + first. Ping @rust-lang/cargo if you want to send a PR for such issue. + +* Dark purple, **Command**-prefixed labels mean the issue has to do with a + specific cargo command. + +* Green, **E**-prefixed labels indicate the level of **experience** or + **effort** necessary to fix the issue. **[E-mentor]** issues also + have some instructions on how to get started. Generally, all of the + **E**-prefixed labels are issues that are ready for someone to contribute + to! + +* Red, **I**-prefixed labels indicate the **importance** of the issue. The + **[I-nominated]** label indicates that an issue has been nominated for + prioritizing at the next triage meeting. + +* Purple gray, **O**-prefixed labels are the **operating system** or platform + that this issue is specific to. + +* Orange, **P**-prefixed labels indicate a bug's **priority**. + +* **S**-prefixed labels are "status" labels, typically used for PRs, but can + also indicate an issue is **[S-blocked]**. + +* The light orange **[relnotes]** label marks issues that should be highlighted + in the [Rust release notes] of the next release. + +* Dark blue, **Z**-prefixed labels are for unstable, [nightly features]. + +[Issue labels]: https://github.com/rust-lang/cargo/labels +[E-easy]: https://github.com/rust-lang/cargo/labels/E-easy +[E-mentor]: https://github.com/rust-lang/cargo/labels/E-mentor +[I-nominated]: https://github.com/rust-lang/cargo/labels/I-nominated +[C-feature-request]: https://github.com/rust-lang/cargo/labels/C-feature-request +[Feature accepted]: https://github.com/rust-lang/cargo/labels/Feature%20accepted +[S-blocked]: https://github.com/rust-lang/cargo/labels/S-blocked +[Rust release notes]: https://github.com/rust-lang/rust/blob/master/RELEASES.md +[nightly features]: https://doc.rust-lang.org/nightly/cargo/reference/unstable.html +[relnotes]: https://github.com/rust-lang/cargo/issues?q=label%3Arelnotes diff --git a/src/doc/contrib/src/process/index.md b/src/doc/contrib/src/process/index.md new file mode 100644 index 00000000000..bb40bf8728a --- /dev/null +++ b/src/doc/contrib/src/process/index.md @@ -0,0 +1,124 @@ +# Process + +This chapter gives an overview of how Cargo comes together, and how you can be +a part of that process. + +See the [Working on Cargo] chapter for an overview of the contribution +process. + +[Working on Cargo]: working-on-cargo.md + +## Cargo team + +Cargo is managed by a [team] of volunteers. The Cargo Team reviews all +changes, and sets the direction for the project. + +The team meets on a weekly basis on a video chat. If you are interested in +participating, feel free to contact us on [Zulip]. + +## Roadmap + +The Cargo team typically establishes a roadmap each year that sets which areas +they will be focusing on. This is usually posted on the Inside Rust Blog (such +as [the 2020 roadmap]). + +The [Roadmap Project Board] is used for tracking major initiatives. This gives +an overview of the things the team is interested in and thinking about. + +The [RFC Project Board] is used for tracking [RFCs]. + +[the 2020 roadmap]: https://blog.rust-lang.org/inside-rust/2020/01/10/cargo-in-2020.html +[Roadmap Project Board]: https://github.com/rust-lang/cargo/projects/1 +[RFC Project Board]: https://github.com/rust-lang/cargo/projects/2 +[RFCs]: https://github.com/rust-lang/rfcs/ + +## Working on small bugs + +Issues labeled with the [E-help-wanted], [E-easy], or [E-mentor] [labels] are +typically issues that the Cargo team wants to see addressed, and are +relatively easy to get started with. If you are interested in one of those, +and it has not already been assigned to someone, leave a comment. See [Issue +assignment](#issue-assignment) below for assigning yourself. + +If there is a specific issue that you are interested in, but it doesn't have +one of the `E-` labels, leave a comment on the issue. If a Cargo team member +has the time to help out, they will respond to help with the next steps. + +[E-help-wanted]: https://github.com/rust-lang/cargo/labels/E-help-wanted +[E-easy]: https://github.com/rust-lang/cargo/labels/E-easy +[E-mentor]: https://github.com/rust-lang/cargo/labels/E-mentor +[labels]: ../issues.md#issue-labels + +## Working on large bugs + +Some issues may be difficult to fix. They may require significant code +changes, or major design decisions. The [E-medium] and [E-hard] [labels] can +be used to tag such issues. These will typically involve some discussion with +the Cargo team on how to tackle it. + +[E-medium]: https://github.com/rust-lang/cargo/labels/E-medium +[E-hard]: https://github.com/rust-lang/cargo/labels/E-hard + +## Working on small features + +Small feature requests are typically managed on the [issue +tracker][issue-feature-request]. Features that the Cargo team have approved +will have the [Feature accepted] label or the [E-mentor] label. If there is a +feature request that you are interested in, feel free to leave a comment +expressing your interest. If a Cargo team member has the time to help out, +they will respond to help with the next steps. Keep in mind that the Cargo +team has limited time, and may not be able to help with every feature request. +Most of them require some design work, which can be difficult. Check out the +[design principles chapter] for some guidance. + +## Working on large features + +Cargo follows the Rust model of evolution. Major features usually go through +an [RFC process]. Therefore, before opening a feature request issue create a +Pre-RFC thread on the [internals][irlo] forum to get preliminary feedback. +Implementing a feature as a [custom subcommand][subcommands] is encouraged as +it helps demonstrate the demand for the functionality and is a great way to +deliver a working solution faster as it can iterate outside of Cargo's release +cadence. + +See the [unstable chapter] for how new major features are typically +implemented. + +[unstable chapter]: unstable.md + +## Bots and infrastructure + +The Cargo project uses several bots: + +* [GitHub Actions] are used to automatically run all tests for each PR. +* [rust-highfive] automatically assigns reviewers for PRs. +* [bors] is used to merge PRs. See [The merging process]. +* [triagebot] is used for assigning issues to non-members, see [Issue + assignment](#issue-assignment). +* [rfcbot] is used for making asynchronous decisions by team members. + +[rust-highfive]: https://github.com/rust-highfive +[bors]: https://buildbot2.rust-lang.org/homu/ +[The merging process]: working-on-cargo.md#the-merging-process +[GitHub Actions]: https://github.com/features/actions +[triagebot]: https://github.com/rust-lang/triagebot/wiki +[rfcbot]: https://github.com/rust-lang/rfcbot-rs + +## Issue assignment + +Normally, if you plan to work on an issue that has been marked with one of the +`E-` tags or [Feature accepted], it is sufficient just to leave a comment that +you are working on it. We also have a bot that allows you to formally "claim" +an issue by entering the text `@rustbot claim` in a comment. See the +[Assignment] docs on how this works. + + +[Assignment]: https://github.com/rust-lang/triagebot/wiki/Assignment +[team]: https://www.rust-lang.org/governance/teams/dev-tools#cargo +[Zulip]: https://rust-lang.zulipchat.com/#narrow/stream/246057-t-cargo +[issue-feature-request]: https://github.com/rust-lang/cargo/labels/C-feature-request +[Feature accepted]: https://github.com/rust-lang/cargo/labels/Feature%20accepted +[design principles chapter]: ../design.md +[RFC process]: https://github.com/rust-lang/rfcs/ +[irlo]: https://internals.rust-lang.org/ +[subcommands]: https://doc.rust-lang.org/cargo/reference/external-tools.html#custom-subcommands diff --git a/src/doc/contrib/src/process/release.md b/src/doc/contrib/src/process/release.md new file mode 100644 index 00000000000..e6fbcaf8f78 --- /dev/null +++ b/src/doc/contrib/src/process/release.md @@ -0,0 +1,109 @@ +# Release process + +Cargo is released with `rustc` using a ["train model"][choochoo]. After a +change lands in Cargo's master branch, it will be synced with the +[rust-lang/rust] repository by a Cargo team member, which happens about once a +week. If there are complications, it can take longer. After it is synced and +merged, the changes will appear in the next nightly release, which is usually +published around 00:30 UTC. + +After changes are in the nightly release, they will make their way to the +stable release anywhere from 6 to 12 weeks later, depending on when during the +cycle it landed. + +The current release schedule is posted on the [Rust Forge]. See the [release +process] for more details on how Rust's releases are created. Rust releases +are managed by the [Release team]. + +[Rust Forge]: https://forge.rust-lang.org/ + +## Build process + +The build process for Cargo is handled as part of building Rust. Every PR on +the [rust-lang/rust] repository creates a full collection of release artifacts +for every platform. The code for this is in the [`dist` bootstrap module]. +Every night at 00:00 UTC, the artifacts from the most recently merged PR are +promoted to the nightly release channel. A similar process happens for beta +and stable releases. + +[`dist` bootstrap module]: https://github.com/rust-lang/rust/blob/master/src/bootstrap/dist.rs + +## Version updates + +Shortly after each major release, a Cargo team member will post a PR to update +Cargo's version in `Cargo.toml`. Cargo's library is permanently unstable, so +its version number starts with a `0`. The minor version is always 1 greater +than the Rust release it is a part of, so cargo 0.49.0 is part of the 1.48 +Rust release. The [CHANGELOG] is also usually updated at this time. + +Also, any version-specific checks that are no longer needed can be removed. +For example, some tests are disabled on stable if they require some nightly +behavior. Once that behavior is available on the new stable release, the +checks are no longer necessary. (I usually search for the word "nightly" in +the testsuite directory, and read the comments to see if any of those nightly +checks can be removed.) + +Sometimes Cargo will have a runtime check to probe `rustc` if it supports a +specific feature. This is usually stored in the [`TargetInfo`] struct. If this +behavior is now stable, those checks should be removed. + +Cargo has several other packages in the [`crates/` directory]. If any of these +packages have changed, the version should be bumped **before the beta +release**. It is rare that these get updated. Bumping these as-needed helps +avoid churning incompatible version numbers. This process should be improved +in the future! + +[`crates/` directory]: https://github.com/rust-lang/cargo/tree/master/crates + +## Docs publishing + +Docs are automatically published during the Rust release process. The nightly +channel's docs appear at . Once +nightly is promoted to beta, those docs will appear at +. Once the stable release is made, it +will appear on (which is the "current" +stable) and the release-specific URL such as +. + +The code that builds the documentation is located in the [`doc` bootstrap +module]. + +[`doc` bootstrap module]: https://github.com/rust-lang/rust/blob/master/src/bootstrap/doc.rs + +## crates.io publishing + +Cargo's library is published to [crates.io] as part of the stable release +process. This is handled by the [Release team] as part of their process. There +is a [`publish.py` script] that in theory should help with this process. The +test and build tool crates aren't published. + +[`publish.py` script]: https://github.com/rust-lang/cargo/blob/master/publish.py + +## Beta backports + +If there is a regression or major problem detected during the beta phase, it +may be necessary to backport a fix to beta. The process is documented in the +[Beta Backporting] page. + +[Beta Backporting]: https://forge.rust-lang.org/release/beta-backporting.html + +## Stable backports + +In (hopefully!) very rare cases, a major regression or problem may be reported +after the stable release. Decisions about this are usually coordinated between +the [Release team] and the Cargo team. There is usually a high bar for making +a stable patch release, and the decision may be influenced by whether or not +there are other changes that need a new stable release. + +The process here is similar to the beta-backporting process. The +[rust-lang/cargo] branch is the same as beta (`rust-1.XX.0`). The +[rust-lang/rust] branch is called `stable`. + +[choochoo]: https://doc.rust-lang.org/book/appendix-07-nightly-rust.html +[rust-lang/rust]: https://github.com/rust-lang/rust/ +[rust-lang/cargo]: https://github.com/rust-lang/cargo/ +[CHANGELOG]: https://github.com/rust-lang/cargo/blob/master/CHANGELOG.md +[release process]: https://forge.rust-lang.org/release/process.html +[`TargetInfo`]: https://github.com/rust-lang/cargo/blob/master/src/cargo/core/compiler/build_context/target_info.rs +[crates.io]: https://crates.io/ +[release team]: https://www.rust-lang.org/governance/teams/operations#release diff --git a/src/doc/contrib/src/process/unstable.md b/src/doc/contrib/src/process/unstable.md new file mode 100644 index 00000000000..522b1176ab6 --- /dev/null +++ b/src/doc/contrib/src/process/unstable.md @@ -0,0 +1,83 @@ +# Unstable features + +Most new features should go through the unstable process. This means that the +feature will only be usable on the nightly channel, and requires a specific +opt-in by the user. Small changes can skip this process, but please consult +with the Cargo team first. + +## Unstable feature opt-in + +For features that require behavior changes or new syntax in `Cargo.toml`, then +it will need a `cargo-features` value placed at the top of `Cargo.toml` to +enable it. The process for doing adding a new feature is described in the +[`features` module]. Code that implements the feature will need to manually +check that the feature is enabled for the current manifest. + +For features that add new command-line flags, config options, or environment +variables, then the `-Z` flags will be needed to enable them. The [`features` +module] also describes how to add these. New flags should use the +`fail_if_stable_opt` method to check if the `-Z unstable-options` flag has +been passed. + +## Unstable documentation + +Every unstable feature should have a section added to the [unstable chapter] +describing how to use the feature. + +`-Z` CLI flags should be documented in the built-in help in the [`cli` +module]. + +[unstable chapter]: https://github.com/rust-lang/cargo/blob/master/src/doc/src/reference/unstable.md +[`cli` module]: https://github.com/rust-lang/cargo/blob/master/src/bin/cargo/cli.rs + +## Tracking issues + +Each unstable feature should get a [tracking issue]. These issues are +typically created when a PR is close to being merged, or soon after it is +merged. Use the [tracking issue template] when creating a tracking issue. + +Larger features should also get a new label in the issue tracker so that when +issues are filed, they can be easily tied together. + +[tracking issue]: https://github.com/rust-lang/cargo/labels/C-tracking-issue +[tracking issue template]: https://github.com/rust-lang/cargo/issues/new?labels=C-tracking-issue&template=tracking_issue.md + +## Stabilization + +After some period of time, typically measured in months, the feature can be +considered to be stabilized. The feature should not have any significant known +bugs or issues, and any design concerns should be resolved. + +The stabilization process depends on the kind of feature. For smaller +features, you can leave a comment on the tracking issue expressing interest in +stabilizing it. It can usually help to indicate that the feature has received +some real-world testing, and has exhibited some demand for broad use. + +For larger features that have not gone through the [RFC process], then an RFC +to call for stabilization might be warranted. This gives the community a final +chance to provide feedback about the proposed design. + +For a small feature, or one that has already gone through the RFC process, a +Cargo Team member may decide to call for a "final comment period" using +[rfcbot]. This is a public signal that a major change is being made, and gives +the Cargo Team members an opportunity to confirm or block the change. This +process can take a few days or weeks, or longer if a concern is raised. + +Once the stabilization has been approved, the person who called for +stabilization should prepare a PR to stabilize the feature. This PR should: + +* Flip the feature to stable in the [`features` module]. +* Remove any unstable checks that aren't automatically handled by the feature + system. +* Move the documentation from the [unstable chapter] into the appropriate + places in the Cargo book and man pages. +* Remove the `-Z` flags and help message if applicable. +* Update all tests to remove nightly checks. +* Tag the PR with [relnotes] label if it seems important enough to highlight + in the [Rust release notes]. + +[`features` module]: https://github.com/rust-lang/cargo/blob/master/src/cargo/core/features.rs +[RFC process]: https://github.com/rust-lang/rfcs/ +[rfcbot]: https://github.com/rust-lang/rfcbot-rs +[Rust release notes]: https://github.com/rust-lang/rust/blob/master/RELEASES.md +[relnotes]: https://github.com/rust-lang/cargo/issues?q=label%3Arelnotes diff --git a/src/doc/contrib/src/process/working-on-cargo.md b/src/doc/contrib/src/process/working-on-cargo.md new file mode 100644 index 00000000000..9ed7e6ca3f7 --- /dev/null +++ b/src/doc/contrib/src/process/working-on-cargo.md @@ -0,0 +1,147 @@ +# Working on Cargo + +This chapter gives an overview of how to build Cargo, make a change, and +submit a Pull Request. + +1. [Check out the Cargo source.](#checkout-out-the-source) +2. [Building Cargo.](#building-cargo) +3. [Making a change.](#making-a-change) +4. [Writing and running tests.](../tests/index.md) +5. [Submitting a Pull Request.](#submitting-a-pull-request) +6. [The merging process.](#the-merging-process) + +## Checkout out the source + +We use the "fork and pull" model [described here][development-models], where +contributors push changes to their personal fork and [create pull requests] to +bring those changes into the source repository. Cargo uses [git] and [GitHub] +for all development. + +1. Fork the [`rust-lang/cargo`] repository on GitHub to your personal account + (see [GitHub docs][how-to-fork]). +2. Clone your fork to your local machine using `git clone` (see [GitHub + docs][how-to-clone]) +3. It is recommended to start a new branch for the change you want to make. + All Pull Requests are made against the master branch. + +## Building Cargo + +Cargo is built by...running `cargo`! There are a few prerequisites that you +need to have installed: + +* `rustc` and `cargo` need to be installed. Cargo is expected to build and + test with the current stable, beta, and nightly releases. It is your choice + which to use. Nightly is recommended, since some nightly-specific tests are + disabled when using the stable release. But using stable is fine if you + aren't working on those. +* A C compiler (typically gcc, clang, or MSVC). +* [git] +* Unix: + * pkg-config + * OpenSSL (`libssl-dev` on Ubuntu, `openssl-devel` on Fedora) +* macOS: + * OpenSSL ([homebrew] is recommended to install the `openssl` package) + +If you can successfully run `cargo build`, you should be good to go! + +[homebrew]: https://brew.sh/ + +## Running Cargo + +You can use `cargo run` to run cargo itself, or you can use the path directly +to the cargo binary, such as `target/debug/cargo`. + +If you are using [`rustup`], beware that running the binary directly can cause +issues with rustup overrides. Usually, when `cargo` is executed as part of +rustup, the toolchain becomes sticky (via an environment variable), and all +calls to `rustc` will use the same toolchain. But when `cargo` is not run via +rustup, the toolchain may change based on the directory. Since Cargo changes +the directory for each compilation, this can cause different calls to `rustc` +to use different versions. There are a few workarounds: + +* Don't use rustup overrides. +* Use `rustup run target/debug/cargo` to execute `cargo`. +* Set the `RUSTC` environment variable to a specific `rustc` executable (not + the rustup wrapper). +* Create a [custom toolchain]. This is a bit of a hack, but you can create a + directory in the rustup `toolchains` directory, and create symlinks for all + the files and directories in there to your toolchain of choice (such as + nightly), except for the `cargo` binary, which you can symlink to your + `target/debug/cargo` binary in your project directory. + +*Normally*, all development is done by running Cargo's test suite, so running +it directly usually isn't required. But it can be useful for testing Cargo on +more complex projects. + +[`rustup`]: https://rust-lang.github.io/rustup/ +[custom toolchain]: https://rust-lang.github.io/rustup/concepts/toolchains.html#custom-toolchains + +## Making a change + +Some guidelines on working on a change: + +* All code changes are expected to comply with the formatting suggested by + `rustfmt`. You can use `rustup component add rustfmt` to install `rustfmt` + and use `cargo fmt` to automatically format your code. +* [Commit as you go][githelp]. +* Include tests that cover all non-trivial code. See the [Testing chapter] for + more about writing and running tests. +* All code should be warning-free. This is checked during tests. + +## Submitting a Pull Request + +After you have committed your work, and pushed it to GitHub, you can +open a Pull Request + +* Push your commits to GitHub and create a pull request against Cargo's + `master` branch. +* Include a clear description of what the change is and why it is being made. +* Use [GitHub's keywords] in the description to automatically link to an issue + if the PR resolves the issue. For example `Closes #1234` will link issue + #1234 to the PR. When the PR is merged, GitHub will automatically close the + issue. + +The [rust-highfive] bot will automatically assign a reviewer for the PR. It +may take at least a few days for someone to respond. If you don't get a +response in over a week, feel free to ping the assigned reviewer. + +When your PR is submitted, GitHub automatically runs all tests. The GitHub +interface will show a green checkmark if it passes, or a red X if it fails. +There are links to the logs on the PR page to diagnose any issues. The tests +typically finish in under 30 minutes. + +The reviewer might point out changes deemed necessary. Large or tricky changes +may require several passes of review and changes. + +## The merging process + +After a reviewer has approved your PR, they will issue a command to the [bors] +bot (also known as "Homu", the software that powers [`@bors`]). Bors will +create a temporary branch with your PR, and run all tests. Only if all tests +pass will it merge the PR to master. If it fails, the bot will leave a comment +on the PR. This system ensures that the master branch is always in a good +state, and that merges are processed one at a time. The [Homu queue +dashboard][homu-cargo] shows the current merge queue. Cargo's queue is rarely +busy, but a busy project like the [rust repo][homu-rust] is constantly full. + +Assuming everything works, congratulations! It may take at least a week for +the changes to arrive on the nightly channel. See the [release chapter] for +more information on how Cargo releases are made. + + +[development-models]: https://help.github.com/articles/about-collaborative-development-models/ +[create pull requests]: https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/creating-a-pull-request +[how-to-fork]: https://docs.github.com/en/github/getting-started-with-github/fork-a-repo +[`rust-lang/cargo`]: https://github.com/rust-lang/cargo/ +[git]: https://git-scm.com/ +[GitHub]: https://github.com/ +[how-to-clone]: https://docs.github.com/en/github/creating-cloning-and-archiving-repositories/cloning-a-repository +[githelp]: https://dont-be-afraid-to-commit.readthedocs.io/en/latest/git/commandlinegit.html +[Testing chapter]: ../tests/index.md +[GitHub's keywords]: https://docs.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue +[rust-highfive]: https://github.com/rust-highfive +[bors]: https://buildbot2.rust-lang.org/homu/ +[`@bors`]: https://github.com/bors +[homu-cargo]: https://buildbot2.rust-lang.org/homu/queue/cargo +[homu-rust]: https://buildbot2.rust-lang.org/homu/queue/rust +[release chapter]: release.md diff --git a/src/doc/contrib/src/tests/index.md b/src/doc/contrib/src/tests/index.md new file mode 100644 index 00000000000..dac047684b4 --- /dev/null +++ b/src/doc/contrib/src/tests/index.md @@ -0,0 +1,20 @@ +# Tests + +Cargo has an extensive test suite. Most of it is implemented as integration +tests in the [`testsuite`] directory. There are several other tests: + +* Unit tests are scattered throughout. +* The dependency resolver has its own set of tests in the [`resolver-tests`] + directory. +* All of the packages in the [`crates`] directory have their own set of tests. +* The [`build-std`] test is for the [build-std feature]. It is separate since + it has some special requirements. +* Documentation has a variety of tests, such as link validation, and the + [SemVer chapter validity checks]. + +[`testsuite`]: https://github.com/rust-lang/cargo/tree/master/tests/testsuite/ +[`resolver-tests`]: https://github.com/rust-lang/cargo/tree/master/crates/resolver-tests +[`crates`]: https://github.com/rust-lang/cargo/tree/master/crates +[`build-std`]: https://github.com/rust-lang/cargo/blob/master/tests/build-std/main.rs +[build-std feature]: https://doc.rust-lang.org/nightly/cargo/reference/unstable.html#build-std +[SemVer chapter validity checks]: https://github.com/rust-lang/cargo/tree/master/src/doc/semver-check diff --git a/src/doc/contrib/src/tests/profiling.md b/src/doc/contrib/src/tests/profiling.md new file mode 100644 index 00000000000..a95d60fbce3 --- /dev/null +++ b/src/doc/contrib/src/tests/profiling.md @@ -0,0 +1,34 @@ +# Profiling + +## Internal profiler + +Cargo has a basic, hierarchical profiler built-in. The environment variable +`CARGO_PROFILE` can be set to an integer which specifies how deep in the +profile stack to print results for. + +```sh +# Output first three levels of profiling info +CARGO_PROFILE=3 cargo generate-lockfile +``` + +## Informal profiling + +The overhead for starting a build should be kept as low as possible +(preferably, well under 0.5 seconds on most projects and systems). Currently, +the primary parts that affect this are: + +* Running the resolver. +* Querying the index. +* Checking git dependencies. +* Scanning the local project. +* Building the unit dependency graph. + +We currently don't have any automated systems or tools for measuring or +tracking the startup time. We informally measure these on changes that are +likely to affect the performance. Usually this is done by measuring the time +for `cargo build` to finish in a large project where the build is fresh (no +actual compilation is performed). [Hyperfine] is a command-line tool that can +be used to roughly measure the difference between different commands and +settings. + +[Hyperfine]: https://github.com/sharkdp/hyperfine diff --git a/src/doc/contrib/src/tests/running.md b/src/doc/contrib/src/tests/running.md new file mode 100644 index 00000000000..b2c4659b4fe --- /dev/null +++ b/src/doc/contrib/src/tests/running.md @@ -0,0 +1,41 @@ +# Running Tests + +Using `cargo test` is usually sufficient for running the full test suite. This +can take a few minutes, so you may want to use more targeted flags to pick the +specific test you want to run, such as `cargo test --test testsuite +-- check::check_success`. + +## Running nightly tests + +Some tests only run on the nightly toolchain, and will be ignored on other +channels. It is recommended that you run tests with both nightly and stable to +ensure everything is working as expected. + +Some of the nightly tests require the `rustc-dev` and `llvm-tools-preview` +rustup components installed. These components include the compiler as a +library. This may already be installed with your nightly toolchain, but if it +isn't, run `rustup component add rustc-dev llvm-tools-preview +--toolchain=nightly`. + +## Running cross tests + +Some tests exercise cross compiling to a different target. This will require +you to install the appropriate target. This typically is the 32-bit target of +your host platform. For example, if your host is a 64-bit +`x86_64-unknown-linux-gnu`, then you should install the 32-bit target with +`rustup target add i686-unknown-linux-gnu`. If you don't have the alternate +target installed, there should be an error message telling you what to do. You +may also need to install additional tools for the target. For example, on Ubuntu +you should install the `gcc-multilib` package. + +If you can't install an alternate target, you can set the +`CFG_DISABLE_CROSS_TESTS=1` environment variable to disable these tests. The +Windows cross tests only support the MSVC toolchain. + +## Running build-std tests + +The `build-std` tests are disabled by default, but you can run them by setting +the `CARGO_RUN_BUILD_STD_TESTS=1` environment variable and running `cargo test +--test build-std`. This requires the nightly channel, and also requires the +`rust-src` component installed with `rustup component add rust-src +--toolchain=nightly`. diff --git a/src/doc/contrib/src/tests/writing.md b/src/doc/contrib/src/tests/writing.md new file mode 100644 index 00000000000..3343f1e6598 --- /dev/null +++ b/src/doc/contrib/src/tests/writing.md @@ -0,0 +1,132 @@ +# Writing Tests + +The following focuses on writing an integration test. However, writing unit +tests is also encouraged! + +## Testsuite + +Cargo has a wide variety of integration tests that execute the `cargo` binary +and verify its behavior, located in the [`testsuite`] directory. The +[`support`] crate contains many helpers to make this process easy. + +These tests typically work by creating a temporary "project" with a +`Cargo.toml` file, executing the `cargo` binary process, and checking the +stdout and stderr output against the expected output. + +### `cargo_test` attribute + +Cargo's tests use the `#[cargo_test]` attribute instead of `#[test]`. This +attribute injects some code which does some setup before starting the test, +creating the little "sandbox" described below. + +### Basic test structure + +The general form of a test involves creating a "project", running `cargo`, and +checking the result. Projects are created with the [`ProjectBuilder`] where +you specify some files to create. The general form looks like this: + +```rust,ignore +let p = project() + .file("src/main.rs", r#"fn main() { println!("hi!"); }"#) + .build(); +``` + +The project creates a mini sandbox under the "cargo integration test" +directory with each test getting a separate directory such as +`/path/to/cargo/target/cit/t123/`. Each project appears as a separate +directory. There is also an empty `home` directory created that will be used +as a home directory instead of your normal home directory. + +If you do not specify a `Cargo.toml` manifest using `file()`, one is +automatically created with a project name of `foo` using `basic_manifest()`. + +To run Cargo, call the `cargo` method and make assertions on the execution: + +```rust,ignore +p.cargo("run --bin foo") + .with_stderr( + "\ +[COMPILING] foo [..] +[FINISHED] [..] +[RUNNING] `target/debug/foo` +", + ) + .with_stdout("hi!") + .run(); +``` + +This uses the [`Execs`] struct to build up a command to execute, along with +the expected output. + +See [`support::lines_match`] for an explanation of the string pattern matching. +Patterns are used to make it easier to match against the expected output. + +Browse the `pub` functions in the [`support`] crate for a variety of other +helpful utilities. + +### Testing Nightly Features + +If you are testing a Cargo feature that only works on "nightly" Cargo, then +you need to call `masquerade_as_nightly_cargo` on the process builder like +this: + +```rust,ignore +p.cargo("build").masquerade_as_nightly_cargo() +``` + +If you are testing a feature that only works on *nightly rustc* (such as +benchmarks), then you should exit the test if it is not running with nightly +rust, like this: + +```rust,ignore +if !is_nightly() { + // Add a comment here explaining why this is necessary. + return; +} +``` + +### Platform-specific Notes + +When checking output, use `/` for paths even on Windows: the actual output +of `\` on Windows will be replaced with `/`. + +Be careful when executing binaries on Windows. You should not rename, delete, +or overwrite a binary immediately after running it. Under some conditions +Windows will fail with errors like "directory not empty" or "failed to remove" +or "access is denied". + +### Specifying Dependencies + +You should not write any tests that use the network such as contacting +crates.io. Typically, simple path dependencies are the easiest way to add a +dependency. Example: + +```rust,ignore +let p = project() + .file("Cargo.toml", r#" + [package] + name = "foo" + version = "1.0.0" + + [dependencies] + bar = {path = "bar"} + "#) + .file("src/lib.rs", "extern crate bar;") + .file("bar/Cargo.toml", &basic_manifest("bar", "1.0.0")) + .file("bar/src/lib.rs", "") + .build(); +``` + +If you need to test with registry dependencies, see +[`support::registry::Package`] for creating packages you can depend on. + +If you need to test git dependencies, see [`support::git`] to create a git +dependency. + +[`testsuite`]: https://github.com/rust-lang/cargo/tree/master/tests/testsuite/ +[`ProjectBuilder`]: https://github.com/rust-lang/cargo/blob/e4b65bdc80f2a293447f2f6a808fa7c84bf9a357/crates/cargo-test-support/src/lib.rs#L225-L231 +[`Execs`]: https://github.com/rust-lang/cargo/blob/e4b65bdc80f2a293447f2f6a808fa7c84bf9a357/crates/cargo-test-support/src/lib.rs#L558-L579 +[`support`]: https://github.com/rust-lang/cargo/blob/master/crates/cargo-test-support/src/lib.rs +[`support::lines_match`]: https://github.com/rust-lang/cargo/blob/e4b65bdc80f2a293447f2f6a808fa7c84bf9a357/crates/cargo-test-support/src/lib.rs#L1322-L1332 +[`support::registry::Package`]: https://github.com/rust-lang/cargo/blob/e4b65bdc80f2a293447f2f6a808fa7c84bf9a357/crates/cargo-test-support/src/registry.rs#L73-L149 +[`support::git`]: https://github.com/rust-lang/cargo/blob/master/crates/cargo-test-support/src/git.rs From dffb8047429eb936bf13b114e48956e5078b1d29 Mon Sep 17 00:00:00 2001 From: Eric Huss Date: Sat, 26 Sep 2020 09:12:06 -0700 Subject: [PATCH 2/3] Add introduction to the architecture chapter. --- src/doc/contrib/src/architecture/index.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/src/doc/contrib/src/architecture/index.md b/src/doc/contrib/src/architecture/index.md index 42e34a2c8e6..fded5fc4b89 100644 --- a/src/doc/contrib/src/architecture/index.md +++ b/src/doc/contrib/src/architecture/index.md @@ -1 +1,8 @@ # Architecture Overview + +This chapter gives a very high-level overview of Cargo's architecture. This is +intended to give you links into the code which is hopefully commented with +more in-depth information. + +If you feel something is missing that would help you, feel free to ask on +Zulip. From b698b254d6ef4cf88f823f73e846900301269c1b Mon Sep 17 00:00:00 2001 From: Eric Huss Date: Sat, 26 Sep 2020 09:13:04 -0700 Subject: [PATCH 3/3] Number the subcommand steps. --- src/doc/contrib/src/architecture/subcommands.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/doc/contrib/src/architecture/subcommands.md b/src/doc/contrib/src/architecture/subcommands.md index 85b0af37158..bdb586c24ed 100644 --- a/src/doc/contrib/src/architecture/subcommands.md +++ b/src/doc/contrib/src/architecture/subcommands.md @@ -7,10 +7,10 @@ subcommands live in [`src/bin/cargo/commands`] directory. Each subcommand, such as [`src/bin/cargo/commands/build.rs`], usually performs the following: -* Parse the CLI flags. See the [`command_prelude`] module for some helpers to make this easier. -* Load the config files. -* Discover and load the workspace. -* Calls the actual implementation of the subcommand which resides in [`src/cargo/ops`]. +1. Parse the CLI flags. See the [`command_prelude`] module for some helpers to make this easier. +2. Load the config files. +3. Discover and load the workspace. +4. Calls the actual implementation of the subcommand which resides in [`src/cargo/ops`]. If the subcommand is not found in the built-in list, then Cargo will automatically search for a subcommand named `cargo-{NAME}` in the users `PATH`