profile.dev/profile.test confusion

[rukai]

This issue applies to #[test] in libraries but not #[test] in binaries.
It also applies to tests in tests/ directory for both libraries and binaries.

When I run cargo test in one of the above cases the tests are compiled seperately to the main project.
The project uses profile.dev and the tests use profile.test

With the following cargo.toml:

[profile.dev]
opt-level = 2

[profile.test]
opt-level = 0

cargo test -v outputs the following

     Running `rustc --crate-name foo lib.rs --emit=dep-info,link -C debuginfo=2 --test -C metadata=d98c9b07f6fdda49 -C extra-filename=-d98c9b07f6fdda49 --out-dir /home/rubic/Foo/foo/target/debug/deps -L dependency=/home/rubic/Foo/foo/target/debug/deps -Zincremental=/home/rubic/Foo/foo/target/debug/incremental`
     Running `rustc --crate-name foo lib.rs --crate-type lib --emit=dep-info,link -C opt-level=2 -C debuginfo=2 -C debug-assertions=on -C metadata=cdf7b1bd72ef34ae -C extra-filename=-cdf7b1bd72ef34ae --out-dir /home/rubic/Foo/foo/target/debug/deps -L dependency=/home/rubic/Foo/foo/target/debug/deps -Zincremental=/home/rubic/Foo/foo/target/debug/incremental`

My reading of http://doc.crates.io/manifest.html had led me to assume that ALL compilation that occurs with cargo test would use profile.test
The current functionality seems to be intended. So maybe just a change to the documentation to make that more obvious would be useful.

The reason this was an issue for me was because I set [profile.dev] opt-level=2 without realizing it would cause my unittests to time out on travis. (Followed by hours of fiddling and research)

[alexcrichton]

cc @matklad

Yeah this is unfortunately working as intended today, it's just not working great to begin with :(. Definitely confusing!

[dwijnand]

I feel like this topic is brought up often. Doesn't this duplicate existing tickets? cc @ehuss

[ehuss]

I don't think there are any other tickets that capture exactly what this is conveying. In the new documentation, I attempted to explain profile selection a little more, such as here, but it's still not terribly clear. And the "profiles" section in manifest.md should be expanded with a clarification on how it works.

I'm often a little wary of making manifest.md even longer, since it's already quite long. What do you think about breaking it up a little? Maybe add dedicated pages for "profiles", "features", and "workspaces"? Maybe even a dedicated page describing targets (lib/bins/examples/tests/benches) and their configuration?

It's tricky with documentation to hit a good balance between concise and complete. I like the brevity of the current way the profiles are shown as a toml document. But some of the options definitely need expanding. And when profile overrides lands, it will only get longer.