Sharing code between integration tests and benchmarks

[MelGrubb]

As I've been adding more tests and more benchmarks to my project, I'm realizing that nearly every benchmark I write is also a test I want to run. Yes, I want to know how fast something runs, and whether changes I've made have had a positive impact, but I also want to know that the answers returned from the code I was calling are correct. It doesn't seem appropriate to put asserts in a benchmark, although I could isolate them in the Cleanup if I wanted so they wouldn't interfere with the timing, but would cause the test to break if something went wrong. That seems like a pretty serious mixing of concerns, though. Ideally, I'd like to build an integration test to prove the code is doing what it should, and then leverage that same code in a benchmark to see how fast it's doing it.

Surely I can't be the only one who has wanted to do something like this, so I'm looking for any advice or lessons learned from those who may have attempted it before. Is this a bad idea in general? Is there some landmine I'm not seeing? How would you break up the code to make it reusable as both a test and a benchmark? It seems to me that the given/when/then style of a traditional test could be turned into a benchmark easily enough. Given is the setup, When is the execution, and Then is the assertions. So, the only real change to turn that into a benchmark is skipping the Then step, right? There's nothing actually stopping me from calling the Given() method, but from a benchmark, after all. It's just a method in a class in a library like any other.

To be clear, I'm talking about pretty course benchmarks here. I'm not tuning at a microscopic level. I'm asking things like "would this part perform better with .Include()s or with Lazy-loading?", or "Does EF8 run this part better than EF7 did?", or "Would it be worthwhile to pull this part out into a stored procedure?". You know, big, broad strokes that drive architectural decisions.

I think I'd also just like the convenience of being able to debug the test code in isolation to verify its operation before unleashing the full fury of BenchmarkDotNet on it to verify its performance.

[mawosoft]

From its early days, BDN still has ReturnValueValidator, but it wasn't even updated when arguments support was added (it only supports parametrized properties). There also aren't any IntroSamples for it. Whether the reason for neglecting this is lack of usage and/or the creators deciding that mixing testing and benchmarking is not a good idea, I don't know.

If your last paragraph refers to the ability to debug a benchmark to manually verify it works correctly, see the BDN Troubleshooting docs. You might also want to take a look at the example code in this class from my BDN extension library, which is intended to make debugging of individual benchmarks easier.

[timcassell]

I think you could use a custom configuration to drive your own custom asserts. Use the ConditionalAttribute to strip out the assert calls when actually benchmarking. As a bonus, you could use the Dry job and/or InProcess toolchain in your test configuration to make it faster.

Coincidentally, BDN recently added BenchmarkDotNet.TestAdapter.

Side note - be careful about values that are only passed to the assert method, as dead code elimination could take effect, spoiling the benchmark.