Something went wrong with outliers when using --launchCount

[AndyAyersMS]

I have been playing around with --launchCount and noticed this when looking at the data:

BenchmarkDotNet v0.13.7-nightly.20230717.35, Windows 11 (10.0.22621.2283/22H2/2022Update/SunValley2)
Intel Core i7-8700 CPU 3.20GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
.NET SDK 8.0.100-preview.7.23376.3
[Host] : .NET 6.0.22 (6.0.2223.42425), X64 RyuJIT AVX2
Job-QYHTSN : .NET 9.0.0 (42.42.42.42424), X64 RyuJIT AVX2

EnvironmentVariables=DOTNET_JitCSEHash=26ab038f,DOTNET_JitDisasmSummary=1,DOTNET_JitStdOutFile=D:\bugs\cse\base.4a.list,DOTNET_JitCSEMask=4 PowerPlanMode=00000000-0000-0000-0000-000000000000 Arguments=/p:EnableUnsafeBinaryFormatterSerialization=true
Toolchain=CoreRun IterationTime=250.0000 ms LaunchCount=100
MaxIterationCount=20 MinIterationCount=15 WarmupCount=1

Method	Count	Mean	Error	StdDev	Median	Min	Max	Gen0	Gen1	Allocated
Dictionary	512	13.71 μs	0.052 μs	0.598 μs	13.82 μs	12.44 μs	17.10 μs	7.6273	0.7133	46.97 KB

// * Hints *
Outliers
TryAddDefaultSize.Dictionary: EnvironmentVariables=DOTNET_JitCSEHash=26ab038f,DOTNET_JitDisasmSummary=1,DOTNET_JitStdOutFile=D:\bugs\cse\base.4a.list,DOTNET_JitCSEMask=4, PowerPlanMode=00000000-0000-0000-0000-000000000000, Arguments=/p:EnableUnsafeBinaryFormatterSerialization=true, Toolchain=CoreRun, IterationTime=250.0000 ms, LaunchCount=100, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1 -> Something went wrong with outliers: Size(WorkloadActual) = 1548, Size(WorkloadActual/Outliers) = 34, Size(Result) = 1467), OutlierMode = RemoveUpper

This was running --filter "System.Collections.TryAddDefaultSize<String>.Dictionary" from dotnet/performance using their config setup and a locally built corerun. But I suspect it will repro with a normal build.

[AndreyAkinshin]

@AndyAyersMS thanks for the report and sorry for the late reply. Could you please share the full log from the run that reproduces this problem?

[AndreyAkinshin]

@AndyAyersMS no logs needed, I have been able to reproduce the problem.

[AndyAyersMS]

@AndyAyersMS no logs needed, I have been able to reproduce the problem.

That's good because I likely don't have it anymore.

[AndreyAkinshin]

@AndyAyersMS it turned out that this check was incorrect, so I just removed it. Thanks again for the report!