Port from bespoke assertions of snapbox

[epage]

Traditionally, cargo-test-support has had bespoke assertions, requiring hand implemented diff algorithms. As there is less of a community around this, the knowledge is more specialized, its less likely to be documented, and we are on our own for feature development.

In #13980 and #14031, we introduced the use of snapbox as replacement for our own assertions and need to work to migrate to them.

Aside: doc updates related to this transition

• feat(test): Add cargo_test to test-support prelude #14243
• fix(test): Move path2url to CargoPathExt::to_url  #14266
• fix(test)!: Clarify extension trait role with rename #14269
• feat(test): Re-export ProcessBuilder #14268
• fix(test): Move 'cargo_home' from 'install' to 'paths'  #14270
• docs(test): Expand documentation of cargo-test-support #14272

Porting Instructions

1. Select a file to port

The files should have #![allow(deprecated)] at the top, e.g.

$ git pull --rebase
$ rg '#!.allow.deprecated' tests

Please check the comments on this issue for anyone to have claimed the file and then post that you claim the file

2. Remove #![allow(deprecated)] to identify what work is needed

If nothing, congrats, that was easy!

3. Resolve basic deprecations

Replace Execs::with_stdout("...") with

use cargo_test_support::prelude::*;
use cargo_test_support::str;

.with_stdout_data(str![])

Replace Execs::with_stderr("...") with

use cargo_test_support::prelude::*;
use cargo_test_support::str;

.with_stderr_data(str![])

(prelude is only needed for some steps)

Side note: If the test is specifically wanting to assert an empty string, use "" rather than str![] so that people are less likely to accidentally do a snapshot update to bless output that gets added.

Commit these changes

Run SNAPSHOTS=overwrite cargo test && cargo check --test testsuite. In rare cases, SNAPSHOTS=overwrite may make bad edits. Feel free to create a reproduction case and create an issue. They are usually obvious how to fix, so don't worry.

Diff the snapshots. Is there anything machine or run specific in them that previously was elided out with [..]?

• Can you auto-redact it like 5ea1c8f? If so, revert the snapshot updates, add the auto-redaction (as a separate commit before any test changes), and re-run the snapshots
• Otherwise, replace it with [..]

Once its working, amend your commit with the snapshots

4. Resolve non-literal deprecations

Like Step 3, but not just with_stdout("...") but with variables or format!.

Evaluate whether a literal could be used

• Is the variable important for showing that output remains unchanged between instances, then keep it
• Are we specifically testing for what we are using format! for, then keep it
  • If not, see if it can be expressed with an auto-redaction

So this can end up with either

• with_std*_data(expression)
• with_std*_data(str![]) like in Step 3

5. Repeat with "straight forward" deprecations

• with_stdout_unordered(expected) -> with_stdout_data(expected.unordered())
• with_stderr_unordered(expected) -> with_stderr_data(expected.unordered())
• with_json(expected) -> with_stdout_data(expected.json_lines()) or with_stdout_data(expected.json())
  • May also require .unordered()

6. Contains / Does not Contains deprecations

A judgement needs to be made for how to handle these.

A contains can be modeled by surrounding a entry with ... (text) or "...", (json lines). The question is whether we should still do this or use a different method.

• If replacing a contains with an equality check, use multi-line globs (...) for rustc errors that weren't previously matched to minimize tying Cargo's tests to the exact output of rustc
• Multiple contains can be merged into a single eq with .. interspersed (or one ... and .unordered()). For example, see 3054936
  • When using unordered, likely string literals, rather than str![] should be used, as these can't correctly be updated

A does_not_contain/line_without cannot be modeled at this time. The challenge with these is that they are brittle and you can't tell when they are no longer testing for what you think because the output changed. We should evaluate what to do with these on a case-by-case basis.

When in doubt, feel free to put #[allow(deprecated)] on a statement and move on. We can come back to these later.

[epage]

#14041 highlights a blocker for tests/testsuite/message_format.rs (and other jsonlines tests): snapbox can only render the output as jsonlines and not our "pretty printed" variant which can make the messages harder to read.

[epage]

I've created assert-rs/snapbox#348 to track what we need from snapbox for jsonlines. See that issue for my proposed solution.

[epage]

I fixed assert-rs/snapbox#348 but found assert-rs/snapbox#351 and assert-rs/snapbox#352

[epage]

Looks like we've finished every file-level migration. We have 124 remaining #[allow(deprecated)]s left.