cargo bolero test --help is wrong, or test searching is broken

[DianaNites]

The help says <test> is the name of the test target, which it's not, it only seems to work if it is the path to the target, like module::tests::test_fn instead of test_fn.

Or if it's intended to be the name and work like cargo test does, then thats it only accepts paths when it shouldn't.

[camshaft]

The help is wrong; the full path to the harness is required. Due to how the fuzzing engines are integrated, currently we only support running a single test at a time so it uses the --exact flag to select that harness.

[DianaNites]

More fun: That isn't true for kani, which does accept fuzzy searching and will run multiple kani harnesses

[camshaft]

Correct. The Kani interface is more like cargo test. This is possible there since Kani proofs actually terminate with an answer. (Note that if you're using bolero with Kani you can invoke cargo kani directly).

The fuzzer engines are a bit more complicated since they can technically run forever and would never yield execution to the next test. One option is to put a timer on each harness but it wasn't obvious what that length would be since it's harness-specific and dependent on if this is your first run or if you're just wanting to update a corpus. There's also some global state/interactions with external binaries that assume it's only running a single harness which was tricky to work around last time I looked at it.

Another thing to note is that cargo test will run bolero harnesses as well. Each harness will try to read the corpus and replay it, in addition to generating 1000 random inputs.

Long story short, I would prefer to have the interfaces all the same. I just haven't had a chance to figure out what it would take to get there.

[DianaNites]

Is it documented anywhere that it tests 1000 random inputs by default? Is there a way to configure that?