Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Windows/x64: 46 Regressions on 9/19/2023 3:42:14 PM #22285

Closed
performanceautofiler bot opened this issue Sep 26, 2023 · 4 comments
Closed

[Perf] Windows/x64: 46 Regressions on 9/19/2023 3:42:14 PM #22285

performanceautofiler bot opened this issue Sep 26, 2023 · 4 comments
Labels
arch-x64 branch-refs/heads/main kind-micro os-windows perf-regression runtime-coreclr

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Sep 26, 2023 •
edited
Loading

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 19445f05aae8a04a2d436e42908429cd90ac1ab8
Compare 1185d19592d2a3f2ae79bd6cb0d7503db09b7d1d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.TryGetValueFalse<Int32, Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
14.05 μs 16.47 μs 1.17 0.02 False
7.35 μs 8.17 μs 1.11 0.13 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.TryGetValueFalse&lt;Int32, Int32&gt;*'

Payloads

Baseline
Compare

System.Collections.TryGetValueFalse<Int32, Int32>.ImmutableSortedDictionary(Size: 512)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.46962220219318 > 14.751057034786177.
IsChangePoint: Marked as a change because one of 6/5/2023 10:43:22 AM, 6/9/2023 2:21:24 AM, 7/14/2023 2:26:40 PM, 7/27/2023 4:03:50 PM, 8/28/2023 1:23:07 PM, 9/1/2023 9:23:50 AM, 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -127.9537245181944 (T) = (0 -16546.412166815015) / Math.Sqrt((5232.22468973843 / (21)) + (3186.039158515527 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.17802518619674046 = (14045.889986643817 - 16546.412166815015) / 14045.889986643817 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Collections.TryGetValueFalse<Int32, Int32>.ImmutableDictionary(Size: 512)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.173014442405174 > 7.725936008041474.
IsChangePoint: Marked as a change because one of 6/22/2023 7:55:14 AM, 7/12/2023 2:12:47 AM, 7/27/2023 4:03:50 PM, 9/4/2023 5:50:19 AM, 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -30.14778974674059 (T) = (0 -8037.469153578371) / Math.Sqrt((4282.269042415115 / (21)) + (6410.5033149096835 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.08862028027998183 = (7383.170513331993 - 8037.469153578371) / 7383.170513331993 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 19445f05aae8a04a2d436e42908429cd90ac1ab8
Compare 1185d19592d2a3f2ae79bd6cb0d7503db09b7d1d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.Span<Char>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
13.26 ns 16.28 ns 1.23 0.21 False
6.26 ns 7.76 ns 1.24 0.16 False
10.04 ns 12.63 ns 1.26 0.10 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Span&lt;Char&gt;*'

Payloads

Baseline
Compare

System.Memory.Span<Char>.IndexOfAnyTwoValues(Size: 512)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.2788104678843 > 13.90909811102365.
IsChangePoint: Marked as a change because one of 5/26/2023 6:22:05 AM, 6/2/2023 1:14:24 PM, 7/23/2023 4:13:29 AM, 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -12.87569297959561 (T) = (0 -15.373494371760648) / Math.Sqrt((0.018207249877837773 / (21)) + (0.6561185170009171 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.16368052947968995 = (13.211095298324286 - 15.373494371760648) / 13.211095298324286 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Memory.Span<Char>.BinarySearch(Size: 33)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.760541901631619 > 6.610798726365816.
IsChangePoint: Marked as a change because one of 5/26/2023 6:22:05 AM, 6/13/2023 11:57:38 AM, 6/19/2023 6:37:24 PM, 6/30/2023 8:34:57 AM, 8/14/2023 10:44:26 PM, 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -17.579556267881678 (T) = (0 -7.585013456268597) / Math.Sqrt((0.0062281619175456274 / (21)) + (0.1288106711760919 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.2112726137724364 = (6.262020101854301 - 7.585013456268597) / 6.262020101854301 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Memory.Span<Char>.BinarySearch(Size: 512)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.632293842607368 > 10.544252864673863.
IsChangePoint: Marked as a change because one of 5/26/2023 6:22:05 AM, 6/13/2023 11:57:38 AM, 6/19/2023 6:37:24 PM, 6/30/2023 8:34:57 AM, 8/14/2023 10:44:26 PM, 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -23.484566178316467 (T) = (0 -12.226518442017783) / Math.Sqrt((0.0010463645950071797 / (21)) + (0.20639452446000967 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.21749233189220665 = (10.042378191421976 - 12.226518442017783) / 10.042378191421976 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 19445f05aae8a04a2d436e42908429cd90ac1ab8
Compare 1185d19592d2a3f2ae79bd6cb0d7503db09b7d1d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Tests.Perf_Encoding

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
10.51 ns 11.70 ns 1.11 0.06 False
19.49 ns 21.57 ns 1.11 0.08 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Tests.Perf_Encoding*'

Payloads

Baseline
Compare

System.Text.Tests.Perf_Encoding.GetByteCount(size: 16, encName: "utf-8")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.695900266964221 > 11.032014625638048.
IsChangePoint: Marked as a change because one of 7/17/2023 6:26:12 PM, 8/27/2023 2:42:29 PM, 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -49.369128763704275 (T) = (0 -11.702991320151014) / Math.Sqrt((0.0038016953150288913 / (21)) + (0.010614442572087272 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.11771710745199988 = (10.47044126114317 - 11.702991320151014) / 10.47044126114317 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Tests.Perf_Encoding.GetByteCount(size: 512, encName: "utf-8")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 21.573933996467648 > 21.299080581278883.
IsChangePoint: Marked as a change because one of 6/6/2023 5:17:50 PM, 6/8/2023 12:15:18 PM, 8/8/2023 9:37:01 AM, 8/23/2023 12:15:00 AM, 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -12.584635952362428 (T) = (0 -21.59631375080929) / Math.Sqrt((0.10750267486382219 / (21)) + (0.19235964016633292 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.071561472827065 = (20.154059564900606 - 21.59631375080929) / 20.154059564900606 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 19445f05aae8a04a2d436e42908429cd90ac1ab8
Compare 1185d19592d2a3f2ae79bd6cb0d7503db09b7d1d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.Span<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
479.77 ns 654.74 ns 1.36 0.13 False
8.91 ns 10.13 ns 1.14 0.04 False
402.30 ns 512.32 ns 1.27 0.20 False
6.93 ns 8.22 ns 1.19 0.12 False
16.36 ns 20.80 ns 1.27 0.09 False
12.38 ns 14.04 ns 1.13 0.10 False

graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Span&lt;Int32&gt;*'

Payloads

Baseline
Compare

System.Memory.Span<Int32>.IndexOfAnyFiveValues(Size: 512)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 654.7385620915032 > 505.59703474533893.
IsChangePoint: Marked as a change because one of 6/13/2023 11:57:38 AM, 6/19/2023 6:37:24 PM, 7/5/2023 7:07:19 PM, 7/12/2023 9:48:15 PM, 8/14/2023 10:44:26 PM, 8/28/2023 1:23:07 PM, 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -39.3644803177278 (T) = (0 -666.3737756947811) / Math.Sqrt((15.871708067298576 / (21)) + (513.2850704179579 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.3849885784214442 = (481.1402679250163 - 666.3737756947811) / 481.1402679250163 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Memory.Span<Int32>.IndexOfAnyTwoValues(Size: 33)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.127882219514358 > 9.296211756781807.
IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 8/19/2023 11:06:43 PM, 9/19/2023 3:42:14 PM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -28.965906522850013 (T) = (0 -10.2316985873909) / Math.Sqrt((0.04288699432462839 / (22)) + (0.006369605818017649 / (23))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (23) - 2, .025) and -0.15417239364828167 = (8.86496561839346 - 10.2316985873909) / 8.86496561839346 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Memory.Span<Int32>.IndexOfAnyFourValues(Size: 512)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 512.3244511479805 > 429.87883361862487.
IsChangePoint: Marked as a change because one of 8/14/2023 10:44:26 PM, 8/28/2023 1:23:07 PM, 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -107.57039949671616 (T) = (0 -513.6288922992748) / Math.Sqrt((12.532905451811171 / (21)) + (8.495677301984662 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.2566196031932785 = (408.73856415581827 - 513.6288922992748) / 408.73856415581827 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Memory.Span<Int32>.ToArray(Size: 4)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.21868140396408 > 7.2708530090477685.
IsChangePoint: Marked as a change because one of 8/25/2023 2:51:11 PM, 8/27/2023 7:52:41 AM, 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -16.9568558834491 (T) = (0 -7.842837530118348) / Math.Sqrt((0.02037682991229816 / (21)) + (0.047569442566976365 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.13311695296711426 = (6.921472236013722 - 7.842837530118348) / 6.921472236013722 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Memory.Span<Int32>.IndexOfValue(Size: 512)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 20.80276138097097 > 17.191738503019234.
IsChangePoint: Marked as a change because one of 5/26/2023 6:22:05 AM, 6/6/2023 4:05:41 PM, 6/13/2023 11:57:38 AM, 6/19/2023 6:37:24 PM, 7/5/2023 7:07:19 PM, 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -37.22822079708861 (T) = (0 -20.509834210535242) / Math.Sqrt((0.008977331798124193 / (21)) + (0.2903345149550477 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.2549251771617121 = (16.343471773291473 - 20.509834210535242) / 16.343471773291473 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Memory.Span<Int32>.ToArray(Size: 33)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.04470882991523 > 12.99429906939141.
IsChangePoint: Marked as a change because one of 6/29/2023 3:17:38 AM, 8/25/2023 2:51:11 PM, 8/27/2023 7:52:41 AM, 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -16.771169546252338 (T) = (0 -13.72382334607316) / Math.Sqrt((0.03621957138605529 / (21)) + (0.10732880962075499 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.10643840121400867 = (12.403603608673631 - 13.72382334607316) / 12.403603608673631 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler
Copy link
Author

performanceautofiler bot commented Sep 26, 2023 •
edited
Loading

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 19445f05aae8a04a2d436e42908429cd90ac1ab8
Compare 1185d19592d2a3f2ae79bd6cb0d7503db09b7d1d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Globalization.Tests.StringSearch

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
303.14 ns 334.26 ns 1.10 0.05 False
18.60 ns 21.16 ns 1.14 0.10 False
302.71 ns 333.85 ns 1.10 0.07 False
303.02 ns 333.62 ns 1.10 0.06 False
443.23 ns 510.24 ns 1.15 0.05 False

graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Globalization.Tests.StringSearch*'

Payloads

Baseline
Compare

System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, None, False))

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 334.2602944225586 > 318.2267386144699.
IsChangePoint: Marked as a change because one of 7/5/2023 7:07:19 PM, 7/12/2023 9:48:15 PM, 7/23/2023 4:13:29 AM, 8/14/2023 10:44:26 PM, 8/27/2023 2:42:29 PM, 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -330.2130041472527 (T) = (0 -334.1933951169257) / Math.Sqrt((0.06416579024464766 / (21)) + (0.14057616899460057 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.10288052681853614 = (303.01867427197095 - 334.1933951169257) / 303.01867427197095 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, Ordinal, False))

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 21.163332906368602 > 19.529193793070128.
IsChangePoint: Marked as a change because one of 8/10/2023 8:47:51 AM, 8/19/2023 11:06:43 PM, 9/2/2023 3:43:46 AM, 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -12.634091835997982 (T) = (0 -20.595529154238136) / Math.Sqrt((0.04959813715719044 / (21)) + (0.5724371183210438 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.11027103551825515 = (18.550001301821318 - 20.595529154238136) / 18.550001301821318 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (, None, False))

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 333.848700822802 > 318.1646620669183.
IsChangePoint: Marked as a change because one of 7/5/2023 7:07:19 PM, 7/12/2023 9:48:15 PM, 7/23/2023 4:13:29 AM, 8/14/2023 10:44:26 PM, 8/27/2023 2:42:29 PM, 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -409.27179588066093 (T) = (0 -334.1884333243668) / Math.Sqrt((0.06355740714827794 / (21)) + (0.06690922741474345 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.10300318545306823 = (302.9804788705991 - 334.1884333243668) / 302.9804788705991 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, IgnoreNonSpace, False))

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 333.6235096595971 > 318.1274349476145.
IsChangePoint: Marked as a change because one of 7/5/2023 7:07:19 PM, 7/12/2023 9:48:15 PM, 7/23/2023 4:13:29 AM, 8/14/2023 10:44:26 PM, 8/27/2023 2:42:29 PM, 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -20.138276538743085 (T) = (0 -334.29984353778514) / Math.Sqrt((45.99288259252624 / (21)) + (0.16603815163882746 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.09804532993137181 = (304.4499479440243 - 334.29984353778514) / 304.4499479440243 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (en-US, IgnoreCase, False))

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 510.23606100360774 > 465.2630573055677.
IsChangePoint: Marked as a change because one of 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -622.061092786205 (T) = (0 -510.4985764579372) / Math.Sqrt((0.10253280448669896 / (21)) + (0.16438150548493186 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.1520518933902027 = (443.1211644083728 - 510.4985764579372) / 443.1211644083728 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 19445f05aae8a04a2d436e42908429cd90ac1ab8
Compare 1185d19592d2a3f2ae79bd6cb0d7503db09b7d1d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.ReadOnlySpan

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
18.56 ns 21.88 ns 1.18 0.16 False
29.72 ns 31.77 ns 1.07 0.03 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.ReadOnlySpan*'

Payloads

Baseline
Compare

System.Memory.ReadOnlySpan.IndexOfString(input: "だだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだxだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだ", value: "x", comparisonType: Ordinal)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 21.884780340870815 > 18.886697504446097.
IsChangePoint: Marked as a change because one of 8/8/2023 9:37:01 AM, 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -11.188384847083906 (T) = (0 -21.677806261775455) / Math.Sqrt((0.08881030794820553 / (21)) + (2.0567612671004536 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.1831156188291738 = (18.322643972216415 - 21.677806261775455) / 18.322643972216415 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Memory.ReadOnlySpan.IndexOfString(input: "foobardzsdzs", value: "rddzs", comparisonType: InvariantCulture)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 31.767400717438004 > 30.860988585560865.
IsChangePoint: Marked as a change because one of 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM, 6/8/2023 8:20:44 PM, 6/22/2023 7:55:14 AM, 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -45.41720946290864 (T) = (0 -32.009348102779846) / Math.Sqrt((0.006878578387074677 / (21)) + (0.07073167312356891 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.08837030475048616 = (29.410346793794723 - 32.009348102779846) / 29.410346793794723 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 19445f05aae8a04a2d436e42908429cd90ac1ab8
Compare 1185d19592d2a3f2ae79bd6cb0d7503db09b7d1d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Buffers.Text.Tests.Utf8ParserTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
10.57 ns 12.15 ns 1.15 0.06 False
49.70 ns 63.94 ns 1.29 0.14 False
11.69 ns 13.23 ns 1.13 0.07 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Text.Tests.Utf8ParserTests*'

Payloads

Baseline
Compare

System.Buffers.Text.Tests.Utf8ParserTests.TryParseUInt64Hex(value: FFFFFFFFFFFFFFFF)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.152673383648244 > 11.095584248410141.
IsChangePoint: Marked as a change because one of 6/9/2023 2:21:24 AM, 7/5/2023 3:16:33 PM, 7/23/2023 4:13:29 AM, 8/10/2023 8:47:51 AM, 9/2/2023 3:43:46 AM, 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -92.55053242351897 (T) = (0 -12.010839322378102) / Math.Sqrt((0.000572469480965907 / (21)) + (0.005353719539532016 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.13884434803727402 = (10.546515283741822 - 12.010839322378102) / 10.546515283741822 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Buffers.Text.Tests.Utf8ParserTests.TryParseSingle(value: -3.4028235E+38)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 63.93905451265089 > 51.36084551433388.
IsChangePoint: Marked as a change because one of 6/8/2023 8:02:22 AM, 7/1/2023 6:36:20 PM, 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -10.095757835471822 (T) = (0 -60.85644160957218) / Math.Sqrt((17.23382836813363 / (21)) + (5.656356262837912 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.20555112967790978 = (50.480182973103226 - 60.85644160957218) / 50.480182973103226 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Buffers.Text.Tests.Utf8ParserTests.TryParseInt32(value: -2147483648)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.225083704475129 > 12.22483279588445.
IsChangePoint: Marked as a change because one of 6/30/2023 8:34:57 AM, 7/13/2023 3:06:28 AM, 8/8/2023 9:37:01 AM, 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -13.025425472039787 (T) = (0 -12.734169971821224) / Math.Sqrt((0.015443677391360178 / (21)) + (0.13235714996373946 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.08798186666036419 = (11.704395415072167 - 12.734169971821224) / 11.704395415072167 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 19445f05aae8a04a2d436e42908429cd90ac1ab8
Compare 1185d19592d2a3f2ae79bd6cb0d7503db09b7d1d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.AddGivenSize<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
1.58 μs 1.73 μs 1.09 0.05 False
2.52 μs 2.89 μs 1.15 0.34 False
255.65 μs 280.09 μs 1.10 0.05 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.AddGivenSize&lt;String&gt;*'

Payloads

Baseline
Compare

System.Collections.AddGivenSize<String>.List(Size: 512)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.7295059505000385 > 1.667557376731873.
IsChangePoint: Marked as a change because one of 9/2/2023 3:43:46 AM, 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -12.424535051219848 (T) = (0 -1728.790396943603) / Math.Sqrt((2332.9066038519404 / (21)) + (41.371685628798375 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.08264293351773706 = (1596.8241637401127 - 1728.790396943603) / 1596.8241637401127 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Collections.AddGivenSize<String>.ICollection(Size: 512)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.889816510753629 > 2.579662753576857.
IsChangePoint: Marked as a change because one of 6/13/2023 11:57:38 AM, 6/21/2023 9:22:33 AM, 6/30/2023 8:34:57 AM, 7/17/2023 6:26:12 PM, 7/23/2023 4:13:29 AM, 9/2/2023 9:37:00 AM, 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -15.361730223666893 (T) = (0 -2805.324577685278) / Math.Sqrt((2580.5628230936427 / (21)) + (9995.271995870511 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.14570198864286024 = (2448.5639420145562 - 2805.324577685278) / 2448.5639420145562 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Collections.AddGivenSize<String>.SortedList(Size: 512)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 280.09060905612245 > 268.7353401639345.
IsChangePoint: Marked as a change because one of 8/22/2023 10:44:27 PM, 8/24/2023 11:06:19 AM, 9/19/2023 12:42:36 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -30.6849573233521 (T) = (0 -277206.2600709446) / Math.Sqrt((2065255.1167401585 / (19)) + (10077063.528531851 / (26))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (26) - 2, .025) and -0.08457545389782352 = (255589.64945656963 - 277206.2600709446) / 255589.64945656963 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 19445f05aae8a04a2d436e42908429cd90ac1ab8
Compare 1185d19592d2a3f2ae79bd6cb0d7503db09b7d1d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsKeyFalse<Int32, Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
7.35 μs 7.85 μs 1.07 0.15 False
2.37 μs 2.62 μs 1.10 0.05 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsKeyFalse&lt;Int32, Int32&gt;*'

Payloads

Baseline
Compare

System.Collections.ContainsKeyFalse<Int32, Int32>.ImmutableDictionary(Size: 512)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.854770447530866 > 7.70873828417309.
IsChangePoint: Marked as a change because one of 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -25.664658356845575 (T) = (0 -8008.839555004419) / Math.Sqrt((3467.6088827913445 / (21)) + (12720.14212838423 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.09228588465127488 = (7332.182597563581 - 8008.839555004419) / 7332.182597563581 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Collections.ContainsKeyFalse<Int32, Int32>.IDictionary(Size: 512)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.6159488566476576 > 2.4888775478550205.
IsChangePoint: Marked as a change because one of 6/5/2023 10:43:22 AM, 6/9/2023 2:21:24 AM, 7/12/2023 9:48:15 PM, 7/23/2023 4:13:29 AM, 7/27/2023 9:50:38 AM, 8/28/2023 1:23:07 PM, 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -55.55387733844247 (T) = (0 -2616.7375947279215) / Math.Sqrt((363.21296533125127 / (21)) + (37.685721199284686 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.10158078096945869 = (2375.43867860969 - 2616.7375947279215) / 2375.43867860969 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Sep 26, 2023 •
edited
Loading

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 19445f05aae8a04a2d436e42908429cd90ac1ab8
Compare 1185d19592d2a3f2ae79bd6cb0d7503db09b7d1d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.CreateAddAndClear<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
3.04 μs 3.40 μs 1.12 0.41 False
2.43 μs 2.76 μs 1.14 0.35 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CreateAddAndClear&lt;String&gt;*'

Payloads

Baseline
Compare

System.Collections.CreateAddAndClear<String>.ICollection(Size: 512)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.4014123944678967 > 3.1863869276309535.
IsChangePoint: Marked as a change because one of 8/15/2023 7:48:35 PM, 9/2/2023 3:43:46 AM, 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -14.728082851559472 (T) = (0 -3566.933551289394) / Math.Sqrt((4114.30619418218 / (21)) + (23541.04922755145 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.16501940784254188 = (3061.6945325356182 - 3566.933551289394) / 3061.6945325356182 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Collections.CreateAddAndClear<String>.Stack(Size: 512)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.75649131462526 > 2.502753342799362.
IsChangePoint: Marked as a change because one of 8/29/2023 4:22:52 PM, 9/2/2023 3:43:46 AM, 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -25.877951980700352 (T) = (0 -2700.737051864785) / Math.Sqrt((164.2447949099015 / (21)) + (3464.1884804155056 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.13403831382739745 = (2381.521875349832 - 2700.737051864785) / 2381.521875349832 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 19445f05aae8a04a2d436e42908429cd90ac1ab8
Compare 1185d19592d2a3f2ae79bd6cb0d7503db09b7d1d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.Span<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
2.71 ns 4.31 ns 1.59 0.24 False
8.55 ns 11.10 ns 1.30 0.29 False
9.54 ns 11.88 ns 1.25 0.13 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Span&lt;Byte&gt;*'

Payloads

Baseline
Compare

System.Memory.Span<Byte>.IndexOfAnyTwoValues(Size: 33)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.3120900368429 > 2.8517728765152706.
IsChangePoint: Marked as a change because one of 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -18.025896570190604 (T) = (0 -4.139152495901787) / Math.Sqrt((1.3010189231218542E-06 / (21)) + (0.14971768910290723 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.524316806722034 = (2.715414851852762 - 4.139152495901787) / 2.715414851852762 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Memory.Span<Byte>.IndexOfAnyTwoValues(Size: 512)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.10429889408567 > 9.105817136031838.
IsChangePoint: Marked as a change because one of 8/10/2023 8:47:51 AM, 8/23/2023 12:15:00 AM, 8/28/2023 1:23:07 PM, 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -12.746729968501448 (T) = (0 -10.280589177560541) / Math.Sqrt((0.005160927682072604 / (21)) + (0.37979276674338236 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.18649174696617463 = (8.66469505906612 - 10.280589177560541) / 8.66469505906612 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Memory.Span<Byte>.SequenceEqual(Size: 512)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.881568031752469 > 11.21472071730283.
IsChangePoint: Marked as a change because one of 6/16/2023 8:27:26 PM, 6/22/2023 7:55:14 AM, 6/26/2023 10:42:28 PM, 7/12/2023 2:12:47 AM, 7/23/2023 4:13:29 AM, 8/14/2023 10:44:26 PM, 8/22/2023 10:44:27 PM, 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -9.970327365090517 (T) = (0 -11.574709397900511) / Math.Sqrt((0.21626264767143724 / (21)) + (0.04708392860623532 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.10543335785259426 = (10.47074372749656 - 11.574709397900511) / 10.47074372749656 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 19445f05aae8a04a2d436e42908429cd90ac1ab8
Compare 1185d19592d2a3f2ae79bd6cb0d7503db09b7d1d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_String

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
15.46 ns 16.50 ns 1.07 0.10 False
35.22 ns 40.06 ns 1.14 0.08 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_String*'

Payloads

Baseline
Compare

System.Tests.Perf_String.GetChars(size: 100)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.49653312882517 > 16.19831257988127.
IsChangePoint: Marked as a change because one of 5/25/2023 10:38:38 PM, 6/12/2023 3:15:58 AM, 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -12.588832399808993 (T) = (0 -16.202354359208254) / Math.Sqrt((0.011531571133142637 / (21)) + (0.08147344284067401 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.051297116075806885 = (15.41177476039032 - 16.202354359208254) / 15.41177476039032 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_String.ToUpper(s: "This is a much longer piece of text that might benefit more from vectorization.")

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 40.05881934584737 > 37.58087706183653.
IsChangePoint: Marked as a change because one of 6/22/2023 12:06:43 AM, 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -13.748451220153905 (T) = (0 -39.839431938801525) / Math.Sqrt((0.123269674778972 / (21)) + (2.096519816970966 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.11777744356125748 = (35.64164956833664 - 39.839431938801525) / 35.64164956833664 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 19445f05aae8a04a2d436e42908429cd90ac1ab8
Compare 1185d19592d2a3f2ae79bd6cb0d7503db09b7d1d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.Perf_Frozen<NotKnownComparable>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
7.98 ns 10.43 ns 1.31 0.03 True

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Perf_Frozen&lt;NotKnownComparable&gt;*'

Payloads

Baseline
Compare

System.Collections.Perf_Frozen<NotKnownComparable>.Contains_True(Count: 4)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.432086743373898 > 8.426729063251765.
IsChangePoint: Marked as a change because one of 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -137.639071835645 (T) = (0 -10.430748803441938) / Math.Sqrt((0.005788634603626227 / (21)) + (0.000506363852836335 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.29418132742323577 = (8.059727475909389 - 10.430748803441938) / 8.059727475909389 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 19445f05aae8a04a2d436e42908429cd90ac1ab8
Compare 1185d19592d2a3f2ae79bd6cb0d7503db09b7d1d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Perf_Convert

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
68.18 ns 72.84 ns 1.07 0.05 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Perf_Convert*'

Payloads

Baseline
Compare

System.Perf_Convert.FromBase64Chars

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 72.84114445914015 > 71.63455764587331.
IsChangePoint: Marked as a change because one of 5/26/2023 9:55:14 AM, 6/5/2023 10:43:22 AM, 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -14.974669843739962 (T) = (0 -73.34316797774702) / Math.Sqrt((3.0996760373544476 / (21)) + (1.1619281560757202 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.09937839634640369 = (66.71330655713311 - 73.34316797774702) / 66.71330655713311 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 19445f05aae8a04a2d436e42908429cd90ac1ab8
Compare 1185d19592d2a3f2ae79bd6cb0d7503db09b7d1d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.Memory<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
28.04 ns 31.54 ns 1.12 0.13 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Memory&lt;Byte&gt;*'

Payloads

Baseline
Compare

System.Memory.Memory<Byte>.ToArray(Size: 512)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 31.54170669877053 > 29.29064434868796.
IsChangePoint: Marked as a change because one of 6/29/2023 3:17:38 AM, 7/3/2023 3:46:39 PM, 9/12/2023 12:17:30 AM, 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -14.869148740415039 (T) = (0 -31.578188427140905) / Math.Sqrt((0.7711041380967578 / (21)) + (0.44197535436823465 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.12430746904415765 = (28.08679057694729 - 31.578188427140905) / 28.08679057694729 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 19445f05aae8a04a2d436e42908429cd90ac1ab8
Compare 1185d19592d2a3f2ae79bd6cb0d7503db09b7d1d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in Benchstone.BenchI.BenchE

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
245.10 ms 323.67 ms 1.32 0.05 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Benchstone.BenchI.BenchE*'

Payloads

Baseline
Compare

Benchstone.BenchI.BenchE.Test

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 323.6702866666667 > 257.304078125.
IsChangePoint: Marked as a change because one of 6/13/2023 11:57:38 AM, 6/20/2023 9:17:03 AM, 6/30/2023 8:34:57 AM, 7/12/2023 2:12:47 AM, 8/28/2023 1:23:07 PM, 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -231.84898722178482 (T) = (0 -323337716.17674) / Math.Sqrt((6223445105.224093 / (21)) + (2728049967440.2407 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.3193770509746868 = (245068470.71341357 - 323337716.17674) / 245068470.71341357 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 19445f05aae8a04a2d436e42908429cd90ac1ab8
Compare 1185d19592d2a3f2ae79bd6cb0d7503db09b7d1d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Threading.Tasks.ValueTaskPerfTest

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
16.44 ns 18.03 ns 1.10 0.07 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Threading.Tasks.ValueTaskPerfTest*'

Payloads

Baseline
Compare

System.Threading.Tasks.ValueTaskPerfTest.CreateAndAwait_FromCompletedValueTaskSource_ConfigureAwait

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 18.026978537415783 > 17.268398055876748.
IsChangePoint: Marked as a change because one of 7/1/2023 6:36:20 PM, 7/5/2023 7:07:19 PM, 8/15/2023 7:48:35 PM, 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -9.91637706692242 (T) = (0 -17.853832678697838) / Math.Sqrt((0.35321706112057005 / (21)) + (0.03687468151649827 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.08137490463373381 = (16.510307944260095 - 17.853832678697838) / 16.510307944260095 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 19445f05aae8a04a2d436e42908429cd90ac1ab8
Compare 1185d19592d2a3f2ae79bd6cb0d7503db09b7d1d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Random

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
400.94 ns 503.09 ns 1.25 0.06 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Random*'

Payloads

Baseline
Compare

System.Tests.Perf_Random.ctor_seeded

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 503.0920466477868 > 421.66092623615185.
IsChangePoint: Marked as a change because one of 6/26/2023 10:42:28 PM, 8/28/2023 1:23:07 PM, 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -41.00151049072388 (T) = (0 -490.4006315847816) / Math.Sqrt((13.395934173287088 / (21)) + (96.63787138500683 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.22036348937450506 = (401.84800336507567 - 490.4006315847816) / 401.84800336507567 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Sep 26, 2023 •
edited
Loading

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 19445f05aae8a04a2d436e42908429cd90ac1ab8
Compare 1185d19592d2a3f2ae79bd6cb0d7503db09b7d1d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
97.78 ns 114.63 ns 1.17 0.34 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple*'

Payloads

Baseline
Compare

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 6, Options: None)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 114.6302900484497 > 102.25996722696826.
IsChangePoint: Marked as a change because one of 6/2/2023 6:53:36 PM, 7/6/2023 4:13:54 AM, 8/8/2023 9:37:01 AM, 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -12.494495346141138 (T) = (0 -112.53344570145038) / Math.Sqrt((5.757113555425164 / (21)) + (25.104142452032498 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.1462241046764824 = (98.1775250078278 - 112.53344570145038) / 98.1775250078278 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 19445f05aae8a04a2d436e42908429cd90ac1ab8
Compare 1185d19592d2a3f2ae79bd6cb0d7503db09b7d1d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Linq.Tests.Perf_Enumerable

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
32.88 ns 35.62 ns 1.08 0.07 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Linq.Tests.Perf_Enumerable*'

Payloads

Baseline
Compare

System.Linq.Tests.Perf_Enumerable.ToArray(input: ICollection)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 35.620296559638554 > 34.680237901657534.
IsChangePoint: Marked as a change because one of 6/12/2023 3:15:58 AM, 9/1/2023 10:04:35 PM, 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -17.003580210269686 (T) = (0 -35.7281528163845) / Math.Sqrt((0.2646047824255034 / (21)) + (0.44932795730464337 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.09197477321520428 = (32.71884451248515 - 35.7281528163845) / 32.71884451248515 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 19445f05aae8a04a2d436e42908429cd90ac1ab8
Compare 1185d19592d2a3f2ae79bd6cb0d7503db09b7d1d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.CtorFromCollection<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
3.68 μs 4.10 μs 1.11 0.20 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CtorFromCollection&lt;String&gt;*'

Payloads

Baseline
Compare

System.Collections.CtorFromCollection<String>.ConcurrentStack(Size: 512)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.100942206242886 > 3.9624173790020403.
IsChangePoint: Marked as a change because one of 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -4.601886402104822 (T) = (0 -4140.732897349299) / Math.Sqrt((31858.999085012918 / (21)) + (55796.427584563426 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.07398296228289952 = (3855.4921658604317 - 4140.732897349299) / 3855.4921658604317 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 19445f05aae8a04a2d436e42908429cd90ac1ab8
Compare 1185d19592d2a3f2ae79bd6cb0d7503db09b7d1d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Tests.Perf_Get

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
91.12 ns 114.17 ns 1.25 0.08 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_Get*'

Payloads

Baseline
Compare

System.Text.Json.Tests.Perf_Get.GetBoolean

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 114.16874286307484 > 95.57246936142548.
IsChangePoint: Marked as a change because one of 6/13/2023 11:57:38 AM, 6/21/2023 9:22:33 AM, 6/30/2023 8:34:57 AM, 7/12/2023 9:48:15 PM, 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -36.693800934407086 (T) = (0 -114.35631055629234) / Math.Sqrt((2.401550569718744 / (21)) + (6.703112834598737 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.252068842034188 = (91.3338841421068 - 114.35631055629234) / 91.3338841421068 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 19445f05aae8a04a2d436e42908429cd90ac1ab8
Compare 1185d19592d2a3f2ae79bd6cb0d7503db09b7d1d
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.ReadOnlyMemory<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
28.37 ns 32.34 ns 1.14 0.14 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.ReadOnlyMemory&lt;Byte&gt;*'

Payloads

Baseline
Compare

System.Memory.ReadOnlyMemory<Byte>.ToArray(Size: 512)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 32.34207795466833 > 29.865164688394167.
IsChangePoint: Marked as a change because one of 6/27/2023 3:42:18 AM, 7/3/2023 3:46:39 PM, 7/12/2023 9:48:15 PM, 7/19/2023 7:56:50 PM, 9/12/2023 12:17:30 AM, 9/19/2023 8:08:41 AM, 9/25/2023 11:21:57 AM falls between 9/16/2023 7:32:07 PM and 9/25/2023 11:21:57 AM.
IsRegressionStdDev: Marked as regression because -15.925008493763517 (T) = (0 -31.906350362677575) / Math.Sqrt((0.5355963067160772 / (21)) + (0.36805615318933926 / (24))) is less than -2.016692199226234 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (21) + (24) - 2, .025) and -0.11218184791726862 = (28.68806969150514 - 31.906350362677575) / 28.68806969150514 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@cincuranet
Copy link

Same as #22151 (comment).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
arch-x64 branch-refs/heads/main kind-micro os-windows perf-regression runtime-coreclr
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@cincuranet