ensure the default order of benchmarks is the same as declared in source code

[adamsitnik]

The story behind this fix is quite long.

In the dotnet/performance repo we have a single type that allows for partitioning benchmarks. The idea is quite simple: it's a filter that takes partition index and count and performs modulo operation to check whether current benchmark belongs to given partition (and should be executed).

Entire logic:

public bool Predicate(BenchmarkCase benchmarkCase)
    => _counter++ % _partitionsCount == _partitionIndex; // will return true only for benchmarks that belong to it’s partition

Some time ago @DrewScoggins has realized that some of the benchmarks in our Reporting System seem to be missing data from time to time. In the example below you can see no new uploads since december:

I believe that it's most likely caused by lack of deterministic order in the reflection APIs (dotnet/runtime#46272). There is no guarantee that the value of _counter will be the same for the same benchmark if we compile and run exactly the same source code multiple times.

Example: let's say that we have 4 benchmarks: A, B, C and D.
Usually they are provided to the filter (which uses their index for filtering) as A, B, C, D.
But sometimes they are ordered as A, B, D, C.

So if we have 2 partitions, and run the process with:
--partition-count 2 --partition-index 0: for ABCD it decides to run benchmark A and C
--partition-count 2 --partition-index 1: for ABDC it decides to run benchmark B and C.

As you can see, benchmark D was never executed because the index changed between two runs of exactly same code.

The types are already sorted by name:

BenchmarkDotNet/src/BenchmarkDotNet/Extensions/ReflectionExtensions.cs

Lines 132 to 138 in c3fb7b9

	internal static Type[] GetRunnableBenchmarks(this Assembly assembly)
	=> assembly
	.GetTypes()
	.Where(type => type.ContainsRunnableBenchmarks())
	.OrderBy(t => t.Namespace)
	.ThenBy(t => t.Name)
	.ToArray();

But we can't sort the methods by names as our users are used to the fact that BDN by default runs the benchmarks in the declaration order.

My proposal is to include the source code line provided by the compiler in the [BenchmarkAttribute] and just use the value for ordering.

I have not added any tests as it's super hard to repro this bug.
I am aware of the fact that the partition filter referenced above is fragile.

[AndreyAkinshin]

In general, I like this approach. However, I guess it makes sense to also use CallerFilePath in order to correctly support benchmark ordering in partial classes.

[adamsitnik]

@AndreyAkinshin good point, I've addressed your feedback, PTAL

[YegorStepanov]

It's a regression.

Currently, this code silently filters wrong methods. After this code will throw friendly message (and exit) if any method is not benchmarkMethod.

BenchmarkRunner.Run(typeof(BenchmarkClass), new MethodInfo[] { BenchmarkClass.PrivateMethod, string.Concat });

[adamsitnik]

@YegorStepanov great catch, I've missed the fact that it's called by

public static BenchmarkRunInfo MethodsToBenchmarks(Type containingType, MethodInfo[] benchmarkMethods, IConfig config = null)

which contains user input

[YegorStepanov]

I'm adding checks for BenchmarkRunner now:

Should be BenchmarkRunner fast exit if any MethodInfo is wrong or it should print error to the console and keep running?

The same question for Type[]. (Note BenchmarkRunner.Run with args != null switches to BenchmarkSwitcher and exit with error)

BenchmarkRunner.Run(new Type[] {typeof(BenchmarkClass), typeof(int)}) 

[adamsitnik]

@YegorStepanov I think that for user provided MethodInfo it's OK to fail fast