Benchmarks not buildable on stable

[kentfredric]

See also: rust-lang/rust#21784

kent@katipo2 rand-0.7.2 $ cargo +stable build --benches
   Compiling rand v0.7.2 (<>/rand-0.7.2)
error[E0554]: #![feature] may not be used on the stable release channel
 --> benches/weighted.rs:9:1
  |
9 | #![feature(test)]
  | ^^^^^^^^^^^^^^^^^

error[E0554]: #![feature] may not be used on the stable release channel
 --> benches/seq.rs:9:1
  |
9 | #![feature(test)]
  | ^^^^^^^^^^^^^^^^^

error[E0554]: #![feature] may not be used on the stable release channel
 --> benches/misc.rs:9:1
  |
9 | #![feature(test)]
  | ^^^^^^^^^^^^^^^^^

error: aborting due to previous error

For more information about this error, try `rustc --explain E0554`.
error: Could not compile `rand`.
warning: build failed, waiting for other jobs to finish...
error[E0554]: #![feature] may not be used on the stable release channel
 --> benches/generators.rs:9:1
  |
9 | #![feature(test)]
  | ^^^^^^^^^^^^^^^^^

error: aborting due to previous error

For more information about this error, try `rustc --explain E0554`.
error: aborting due to previous error

error: Could not compile `rand`.
warning: build failed, waiting for other jobs to finish...
For more information about this error, try `rustc --explain E0554`.
error: Could not compile `rand`.
warning: build failed, waiting for other jobs to finish...
error: aborting due to previous error

For more information about this error, try `rustc --explain E0554`.
error: Could not compile `rand`.

To learn more, run the command again with --verbose.

[dhardy]

They never were. Does Rust support benchmarks on stable yet?

[vks]

No, but there are (slightly more clunky to use) libraries that work on stable. bencher is a port of libtest, criterion has more advanced statistics / reports / comparisons but also more dependencies. I suggest to use the latter.

[dhardy]

Switching the benchmark lib is an option but I see little benefit. In my opinion sticking with the status quo is not an issue (though I presume @kentfredric disagrees).

I'll leave this open for a week or so for further input but if nothing happens it will be closed.

[kentfredric]

Part of this bug is acknowledging its not your fault, just a circumstance you've found yourself in due to design decision in rust ( hence the x-ref ).

I'm working on linux-vendoring for Rust stuff, and on the list of "nice to have"'s is:

• Tests runnable with the system (stable) rust
• Examples buildable/runnable with system (stable) rust
• Benchmarks buildable/runnable with system(stable) rust.

IMO, the net utility of benchmarks is lessened if only developers can run them on bleading edge stuff, because it means stable users can't compare the performance of 2 different versions of a package, and can't compare the performance of the same version of a package on 2 different stable rusts.

By all means, if you think the status quo can be improved on, be my guest.

But improvements to the rust ecosystem that allow you to ship usable benchmarks without needing painful contortions would be beneficial to all (and hence, this is a nudge that rust-lang/rust#21784 probably should have a different conclusion than Not A Bug)

[dhardy]

Well, you linked the wrong Rust bug; that one complains about ergonomics. See here and the linked internals post. Summary is that work is still on-going in this area; i.e. support for a benchmarking on stable with the standard libraries is planned but not there yet.

In the mean-time, I don't have anything much against switching to criterion or bencher (it would depend on PR review) but I'm not doing the work myself.

As for Not A Bug: benchmarks-on-stable are a feature we have never delivered until now. It's not broken if it never existed.

From my POV benchmarks are a development tool and not suited for much else (if you wanted to compare to another crate, you should write your own benches doing an apples-to-apples comparison and not assume that you can compare between one project's benches and another).

[kentfredric]

Fair enough, glad to see there's more progress headed in that direction then :) ( gosh finding the right bug is hard :( )

I suspect the ideal approach is to wait for that internals work to finish, and then revisit this situation (ie: perform much fewer changes to use the stabilized feature)

[vks]

This would be fixed by #1039.

[dhardy]

(Refer to #1039 for further discussion.)