Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Windows/x86: 38 Regressions on 4/17/2023 11:29:15 PM #16941

Closed
performanceautofiler bot opened this issue Apr 25, 2023 · 3 comments
Closed

[Perf] Windows/x86: 38 Regressions on 4/17/2023 11:29:15 PM #16941

performanceautofiler bot opened this issue Apr 25, 2023 · 3 comments
Assignees
@kunalspathak
Labels
arch-x86 branch-refs/heads/main kind-micro os-windows perf-regression runtime-coreclr untriaged

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Apr 25, 2023 •
edited
Loading

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7dbf6a5f2fd636db350f9cc31b2dfd3af0eff1a8
Compare eb71e484d2d4a7ae002ea524d2edead773289bbb
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Linq.Tests.Perf_Enumerable

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SingleWithPredicate_LastElementMatches - Duration of single invocation 677.56 ns 735.88 ns 1.09 0.04 False Trace Trace
GroupBy - Duration of single invocation 2.88 μs 3.11 μs 1.08 0.00 True 20600.523025896164 23817.997386699677 1.1561841103140413) Trace Trace
ToDictionary - Duration of single invocation 1.17 μs 1.45 μs 1.24 0.00 True 10368.944049804453 13673.720553738394 1.3187187131168159) Trace Trace
ToDictionary - Duration of single invocation 1.25 μs 1.97 μs 1.58 0.03 False 10971.09497365798 14362.84046692607 1.3091528695551173) Trace Trace
SingleWithPredicate_LastElementMatches - Duration of single invocation 664.53 ns 742.61 ns 1.12 0.01 False Trace Trace
SequenceEqual - Duration of single invocation 986.50 ns 1.08 μs 1.09 0.02 False Trace Trace
ToDictionary - Duration of single invocation 2.62 μs 3.48 μs 1.33 0.00 True Trace Trace
SingleWithPredicate_LastElementMatches - Duration of single invocation 907.62 ns 966.33 ns 1.06 0.01 False Trace Trace
Where - Duration of single invocation 1.15 μs 1.28 μs 1.12 0.01 False Trace Trace

graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Linq.Tests.Perf_Enumerable*'

Payloads

Baseline
Compare

Histogram

System.Linq.Tests.Perf_Enumerable.SingleWithPredicate_LastElementMatches(input: IEnumerable)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 735.8833720030937 > 711.5460096271499.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/13/2023 6:54:14 PM, 2/24/2023 9:01:57 AM, 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -5.42740703885092 (T) = (0 -713.1596429954153) / Math.Sqrt((183.0931320546734 / (16)) + (1010.89563246983 / (33))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (33) - 2, .025) and -0.05192925396262531 = (677.9539976751646 - 713.1596429954153) / 677.9539976751646 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

System.Linq.Tests.Perf_Enumerable.GroupBy(input: IEnumerable)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.107841942011677 > 3.040210537104614.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -66.4006201461177 (T) = (0 -3119.9569447861836) / Math.Sqrt((107.19820383247433 / (12)) + (64.6281808266485 / (36))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (36) - 2, .025) and -0.07493290901860493 = (2902.4666736035183 - 3119.9569447861836) / 2902.4666736035183 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

System.Linq.Tests.Perf_Enumerable.ToDictionary(input: Array)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4517851329985119 > 1.2269382919750111.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -84.9008860569687 (T) = (0 -1460.7016889183753) / Math.Sqrt((3.0181363506771643 / (12)) + (431.9625356211111 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.2511336984242194 = (1167.5024745621536 - 1460.7016889183753) / 1167.5024745621536 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

System.Linq.Tests.Perf_Enumerable.ToDictionary(input: List)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.9699900630914824 > 1.3101979048146752.
IsChangePoint: Marked as a change because one of 2/24/2023 9:01:57 AM, 3/6/2023 11:14:52 AM, 3/17/2023 9:25:32 PM, 3/25/2023 12:18:49 AM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -474.09063686267916 (T) = (0 -1959.3162882407314) / Math.Sqrt((16.63887239291707 / (12)) + (31.068080508828228 / (36))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (36) - 2, .025) and -0.5696536736024993 = (1248.2475091106694 - 1959.3162882407314) / 1248.2475091106694 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

System.Linq.Tests.Perf_Enumerable.SingleWithPredicate_LastElementMatches(input: Array)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 742.6147843582196 > 697.6301032387627.
IsChangePoint: Marked as a change because one of 3/22/2023 7:51:01 PM, 3/25/2023 12:18:49 AM, 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -37.293328716371875 (T) = (0 -743.0109466616406) / Math.Sqrt((62.80984685384855 / (16)) + (0.8576340974875277 / (33))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (33) - 2, .025) and -0.11083366568244074 = (668.8768711426942 - 743.0109466616406) / 668.8768711426942 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

System.Linq.Tests.Perf_Enumerable.SequenceEqual(input1: IEnumerable, input2: IEnumerable)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.076927663277774 > 1035.1858411765786.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/16/2023 6:27:39 AM, 3/20/2023 8:31:51 AM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -26.95154243188936 (T) = (0 -1080.277768439237) / Math.Sqrt((8.148394761602765 / (12)) + (437.2669128837195 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.09672770556797201 = (985.000892158354 - 1080.277768439237) / 985.000892158354 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

System.Linq.Tests.Perf_Enumerable.ToDictionary(input: IEnumerable)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.4775547756981586 > 2.7519925013021247.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -111.227286201351 (T) = (0 -3543.9022231338345) / Math.Sqrt((85.6994450095476 / (12)) + (2261.600777592973 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.35010604679342766 = (2624.906563118347 - 3543.9022231338345) / 2624.906563118347 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

System.Linq.Tests.Perf_Enumerable.SingleWithPredicate_LastElementMatches(input: List)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 966.3335403726705 > 950.0915755503282.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -20.62345261029382 (T) = (0 -980.6026991907164) / Math.Sqrt((65.01676840800393 / (12)) + (260.1666913118044 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.08015527588179348 = (907.8349391851958 - 980.6026991907164) / 907.8349391851958 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

System.Linq.Tests.Perf_Enumerable.Where(input: IEnumerable)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.2777804120340093 > 1.2025407150817966.
IsChangePoint: Marked as a change because one of 2/24/2023 9:01:57 AM, 3/25/2023 12:18:49 AM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -92.37737191647477 (T) = (0 -1282.7757276412804) / Math.Sqrt((2.2438723234667597 / (12)) + (74.1499228929735 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.11931421034064144 = (1146.0372036649946 - 1282.7757276412804) / 1146.0372036649946 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7dbf6a5f2fd636db350f9cc31b2dfd3af0eff1a8
Compare eb71e484d2d4a7ae002ea524d2edead773289bbb
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_SByte

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryParse - Duration of single invocation 12.01 ns 14.81 ns 1.23 0.51 False 169.33279627999434 187.9427187577103 1.1099014655551083) Trace Trace
Parse - Duration of single invocation 12.37 ns 16.20 ns 1.31 0.51 False 167.5578089867756 186.10584629752802 1.1106963466693236) Trace Trace
Parse - Duration of single invocation 12.28 ns 15.88 ns 1.29 0.49 False Trace Trace

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_SByte*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_SByte.TryParse(value: "-128")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.8116109582275 > 12.805772905108872.
IsChangePoint: Marked as a change because one of 2/16/2023 9:09:51 PM, 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -3.7856901883155154 (T) = (0 -13.952397332196998) / Math.Sqrt((1.0027785482016423 / (16)) + (0.8808635485238951 / (33))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (33) - 2, .025) and -0.08827157597895341 = (12.82069442973932 - 13.952397332196998) / 12.82069442973932 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

System.Tests.Perf_SByte.Parse(value: "-128")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.199225205836296 > 13.02378653481448.
IsChangePoint: Marked as a change because one of 2/16/2023 9:09:51 PM, 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -9.97695196763462 (T) = (0 -16.361792462362743) / Math.Sqrt((1.3604963276882729 / (16)) + (0.35018724675812724 / (33))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (33) - 2, .025) and -0.23240567621988056 = (13.27630404344514 - 16.361792462362743) / 13.27630404344514 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

System.Tests.Perf_SByte.Parse(value: "127")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.877581060414135 > 12.512093986367589.
IsChangePoint: Marked as a change because one of 2/16/2023 4:50:02 PM, 3/9/2023 1:54:27 PM, 3/15/2023 5:33:41 AM, 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -11.788108456581782 (T) = (0 -15.97989042844651) / Math.Sqrt((1.5181148175611097 / (16)) + (0.14102210555701664 / (33))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (33) - 2, .025) and -0.3025737179473486 = (12.26793555579204 - 15.97989042844651) / 12.26793555579204 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7dbf6a5f2fd636db350f9cc31b2dfd3af0eff1a8
Compare eb71e484d2d4a7ae002ea524d2edead773289bbb
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.CtorDefaultSize<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Queue - Duration of single invocation 15.58 ns 17.69 ns 1.13 0.19 False Trace Trace

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CtorDefaultSize&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CtorDefaultSize<String>.Queue

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.68613858261634 > 16.334450313078378.
IsChangePoint: Marked as a change because one of 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -10.442294418312175 (T) = (0 -18.591047175590283) / Math.Sqrt((0.5544130316072805 / (16)) + (1.3333485779617154 / (33))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (33) - 2, .025) and -0.1818657705391982 = (15.73025265560281 - 18.591047175590283) / 15.73025265560281 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler
Copy link
Author

performanceautofiler bot commented Apr 25, 2023 •
edited
Loading

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7dbf6a5f2fd636db350f9cc31b2dfd3af0eff1a8
Compare eb71e484d2d4a7ae002ea524d2edead773289bbb
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_UInt32

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryParse - Duration of single invocation 21.38 ns 25.06 ns 1.17 0.47 False Trace Trace
ParseSpan - Duration of single invocation 14.43 ns 15.53 ns 1.08 0.44 False Trace Trace
Parse - Duration of single invocation 22.27 ns 23.42 ns 1.05 0.55 False 252.9661952575253 330.18457907212326 1.3052517896155567) Trace Trace
TryParseHex - Duration of single invocation 13.03 ns 14.38 ns 1.10 0.00 True Trace Trace
Parse - Duration of single invocation 14.24 ns 16.74 ns 1.18 0.43 False Trace Trace

graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_UInt32*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_UInt32.TryParse(value: "4294967295")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 25.06469674162867 > 22.455348223017594.
IsChangePoint: Marked as a change because one of 3/15/2023 5:33:41 AM, 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -7.017606654227718 (T) = (0 -24.04049478308461) / Math.Sqrt((1.923446597207185 / (16)) + (1.4238403485061257 / (33))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (33) - 2, .025) and -0.13376563276775286 = (21.20411316790126 - 24.04049478308461) / 21.20411316790126 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

System.Tests.Perf_UInt32.ParseSpan(value: "12345")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.530792711809887 > 14.855608713633186.
IsChangePoint: Marked as a change because one of 2/16/2023 4:50:02 PM, 3/10/2023 12:53:36 AM, 3/15/2023 5:33:41 AM, 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -2.840085014125083 (T) = (0 -15.654139950363543) / Math.Sqrt((1.6781956775590077 / (16)) + (0.32694507763179836 / (33))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (33) - 2, .025) and -0.0654960017188197 = (14.691880518660652 - 15.654139950363543) / 14.691880518660652 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

System.Tests.Perf_UInt32.Parse(value: "4294967295")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 23.42248822518639 > 23.194175073878817.
IsChangePoint: Marked as a change because one of 2/16/2023 4:50:02 PM, 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -6.328385300991019 (T) = (0 -24.635504674870962) / Math.Sqrt((2.7344601984268038 / (16)) + (3.0146286196941747 / (33))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (33) - 2, .025) and -0.1514781479579189 = (21.394678412752018 - 24.635504674870962) / 21.394678412752018 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

System.Tests.Perf_UInt32.TryParseHex(value: "FFFFFFFF")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.383317578442773 > 13.674735966083317.
IsChangePoint: Marked as a change because one of 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -184.54231044345943 (T) = (0 -14.393947076500343) / Math.Sqrt((0.0003601153337394153 / (16)) + (0.0010566359336485562 / (33))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (33) - 2, .025) and -0.10457157322094064 = (13.031248880077063 - 14.393947076500343) / 13.031248880077063 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

System.Tests.Perf_UInt32.Parse(value: "12345")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.739011817472715 > 14.555096225286933.
IsChangePoint: Marked as a change because one of 2/16/2023 4:50:02 PM, 3/14/2023 10:43:28 AM, 3/15/2023 5:33:41 AM, 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -5.7557218696215 (T) = (0 -15.78131086152257) / Math.Sqrt((0.8907889338196507 / (16)) + (1.1515899543443529 / (33))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (33) - 2, .025) and -0.1232948737102198 = (14.049125684511695 - 15.78131086152257) / 14.049125684511695 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7dbf6a5f2fd636db350f9cc31b2dfd3af0eff1a8
Compare eb71e484d2d4a7ae002ea524d2edead773289bbb
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in Benchmark.GetChildKeysTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
AddChainedConfigurationNoDelimiter - Duration of single invocation 822.32 μs 877.24 μs 1.07 0.17 False 8837485.970819304 9310386.473429952 1.0535107500223626) Trace Trace

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Benchmark.GetChildKeysTests*'

Payloads

Baseline
Compare

Histogram

Benchmark.GetChildKeysTests.AddChainedConfigurationNoDelimiter

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 877.2390972222221 > 866.423673828125.
IsChangePoint: Marked as a change because one of 2/18/2023 3:00:06 AM, 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -13.42345515398165 (T) = (0 -874562.0793472492) / Math.Sqrt((107112541.73884231 / (16)) + (151578384.27005422 / (33))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (33) - 2, .025) and -0.054371634786260094 = (829462.8293225458 - 874562.0793472492) / 829462.8293225458 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7dbf6a5f2fd636db350f9cc31b2dfd3af0eff1a8
Compare eb71e484d2d4a7ae002ea524d2edead773289bbb
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.AddGivenSize<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
HashSet - Duration of single invocation 5.99 μs 6.41 μs 1.07 0.01 False Trace Trace
IDictionary - Duration of single invocation 8.66 μs 11.86 μs 1.37 0.12 True 53411.09154929577 71525.56968733438 1.3391519928275544) Trace Trace
Dictionary - Duration of single invocation 8.44 μs 11.66 μs 1.38 0.14 True 50230.72002923976 67930.19266297176 1.3523635063050854) Trace Trace

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.AddGivenSize&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.AddGivenSize<Int32>.HashSet(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.406543109300746 > 6.271093335815614.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -16.855359685996707 (T) = (0 -6400.856221545331) / Math.Sqrt((7930.236287055452 / (16)) + (282.5888048914817 / (33))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (33) - 2, .025) and -0.06284529168914556 = (6022.378112408682 - 6400.856221545331) / 6022.378112408682 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

System.Collections.AddGivenSize<Int32>.IDictionary(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.855249688144049 > 8.892077712834668.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -58.27712872286452 (T) = (0 -11912.944692359753) / Math.Sqrt((39130.38336729331 / (12)) + (3905.1342672232486 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.3963218587387181 = (8531.66096183625 - 11912.944692359753) / 8531.66096183625 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

System.Collections.AddGivenSize<Int32>.Dictionary(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.659230194319882 > 8.864971962216835.
IsChangePoint: Marked as a change because one of 2/16/2023 6:27:39 AM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -251.30005641563665 (T) = (0 -11592.170115031313) / Math.Sqrt((711.2986448109984 / (12)) + (3539.2046475147617 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.36954880936669865 = (8464.225616312075 - 11592.170115031313) / 8464.225616312075 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7dbf6a5f2fd636db350f9cc31b2dfd3af0eff1a8
Compare eb71e484d2d4a7ae002ea524d2edead773289bbb
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Version

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Parse3 - Duration of single invocation 35.27 ns 38.25 ns 1.08 0.01 True 447.6657868690201 471.82318438353616 1.0539630193396579) Trace Trace
TryParse2 - Duration of single invocation 31.64 ns 33.68 ns 1.06 0.01 False Trace Trace
TryParse4 - Duration of single invocation 56.48 ns 61.74 ns 1.09 0.00 True 682.5620435216961 712.1430849889173 1.043338245582185) Trace Trace
TryParse3 - Duration of single invocation 35.30 ns 37.66 ns 1.07 0.01 False 441.1414055694073 459.92302124590776 1.0425750460949317) Trace Trace
Parse2 - Duration of single invocation 30.77 ns 35.35 ns 1.15 0.02 True 362.727643843694 380.97123372547105 1.0502955597440997) Trace Trace

graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Version*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Version.Parse3

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 38.24724593347506 > 37.149157763846645.
IsChangePoint: Marked as a change because one of 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -21.56481183846607 (T) = (0 -38.17193205078802) / Math.Sqrt((0.2690661235370305 / (16)) + (0.04334516537514495 / (33))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (33) - 2, .025) and -0.08233071448313664 = (35.26827016917551 - 38.17193205078802) / 35.26827016917551 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

System.Tests.Perf_Version.TryParse2

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 33.67818227997316 > 32.12812681773893.
IsChangePoint: Marked as a change because one of 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -16.669595266332074 (T) = (0 -33.26610542313931) / Math.Sqrt((0.18767572370677207 / (16)) + (0.3681887604959654 / (33))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (33) - 2, .025) and -0.08202669259220875 = (30.744255803379286 - 33.26610542313931) / 30.744255803379286 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

System.Tests.Perf_Version.TryParse4

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 61.73816097018008 > 59.30365014027097.
IsChangePoint: Marked as a change because one of 2/6/2023 2:52:08 AM, 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -42.49167517032636 (T) = (0 -61.452177595004294) / Math.Sqrt((0.1455901158378972 / (16)) + (0.17618695540682977 / (31))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (31) - 2, .025) and -0.09178749518197546 = (56.2858412156128 - 61.452177595004294) / 56.2858412156128 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

System.Tests.Perf_Version.TryParse3

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 37.65801580888753 > 37.1032503132986.
IsChangePoint: Marked as a change because one of 3/17/2023 1:30:52 PM, 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -16.60218397360279 (T) = (0 -37.61278728880096) / Math.Sqrt((0.2293693983529111 / (16)) + (0.10247847858038822 / (33))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (33) - 2, .025) and -0.06190125853832021 = (35.42023044644847 - 37.61278728880096) / 35.42023044644847 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

System.Tests.Perf_Version.Parse2

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 35.34777548123096 > 31.976438227205136.
IsChangePoint: Marked as a change because one of 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -62.2838646473998 (T) = (0 -35.26207154537489) / Math.Sqrt((0.06202123268102507 / (16)) + (0.06607870789701481 / (33))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (33) - 2, .025) and -0.15664181887807588 = (30.486595737631667 - 35.26207154537489) / 30.486595737631667 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Apr 25, 2023 •
edited
Loading

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7dbf6a5f2fd636db350f9cc31b2dfd3af0eff1a8
Compare eb71e484d2d4a7ae002ea524d2edead773289bbb
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.IterateForEachNonGeneric<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ArrayList - Duration of single invocation 3.05 μs 3.48 μs 1.14 0.05 False Trace Trace

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IterateForEachNonGeneric&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.IterateForEachNonGeneric<String>.ArrayList(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.4833993279869317 > 3.198837945979177.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/24/2023 9:01:57 AM, 3/6/2023 11:14:52 AM, 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -41.58829524012257 (T) = (0 -3496.7143770648267) / Math.Sqrt((1316.6238680661108 / (16)) + (1157.5223838251043 / (33))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (33) - 2, .025) and -0.14790661995455365 = (3046.1662266598514 - 3496.7143770648267) / 3046.1662266598514 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7dbf6a5f2fd636db350f9cc31b2dfd3af0eff1a8
Compare eb71e484d2d4a7ae002ea524d2edead773289bbb
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Int32

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Parse - Duration of single invocation 13.59 ns 16.61 ns 1.22 0.51 False 184.85987235191345 209.5240670093525 1.1334210304467072) Trace Trace
TryParseSpan - Duration of single invocation 8.79 ns 11.00 ns 1.25 0.54 False 117.69946927397005 129.93569128546918 1.1039615733781838) Trace Trace
ParseHex - Duration of single invocation 16.15 ns 19.93 ns 1.23 0.47 False Trace Trace
ToString - Duration of single invocation 10.69 ns 12.85 ns 1.20 0.05 True Trace Trace
TryParseSpan - Duration of single invocation 13.19 ns 15.72 ns 1.19 0.45 False 181.58085912355827 201.96935636405925 1.112283295380971) Trace Trace
ParseHex - Duration of single invocation 15.05 ns 19.03 ns 1.26 0.47 False Trace Trace
ParseSpan - Duration of single invocation 21.07 ns 23.58 ns 1.12 0.60 False Trace Trace

graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int32*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Int32.Parse(value: "12345")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.614371104762924 > 14.169761375208738.
IsChangePoint: Marked as a change because one of 2/16/2023 1:21:52 PM, 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -9.487779136933119 (T) = (0 -16.819881542233972) / Math.Sqrt((1.1943093336022086 / (16)) + (0.5131007716570343 / (31))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (31) - 2, .025) and -0.20532032312439757 = (13.954698364857855 - 16.819881542233972) / 13.954698364857855 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

System.Tests.Perf_Int32.TryParseSpan(value: "4")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.998361712216154 > 9.113866360228998.
IsChangePoint: Marked as a change because one of 2/16/2023 1:00:08 AM, 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -5.733175655543632 (T) = (0 -10.456511588170814) / Math.Sqrt((0.6750543756776702 / (16)) + (0.44667662244265416 / (33))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (33) - 2, .025) and -0.14867450446155633 = (9.103111061973406 - 10.456511588170814) / 9.103111061973406 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

System.Tests.Perf_Int32.ParseHex(value: "80000000")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 19.93266595237851 > 16.230090995408652.
IsChangePoint: Marked as a change because one of 2/14/2023 11:21:41 AM, 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -25.92926552771014 (T) = (0 -19.18956774747775) / Math.Sqrt((0.23656780200843355 / (16)) + (0.24435361435278502 / (33))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (33) - 2, .025) and -0.25200643495575153 = (15.327052011642374 - 19.18956774747775) / 15.327052011642374 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

System.Tests.Perf_Int32.ToString(value: 2147483647)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.84762368919414 > 11.227056386560434.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -23.25470262806513 (T) = (0 -12.973710281651371) / Math.Sqrt((0.0900622805444348 / (12)) + (0.040348743748696125 / (36))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (36) - 2, .025) and -0.199725171520321 = (10.81390187488587 - 12.973710281651371) / 10.81390187488587 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

System.Tests.Perf_Int32.TryParseSpan(value: "12345")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.71788042647833 > 13.846259516123313.
IsChangePoint: Marked as a change because one of 2/16/2023 9:09:51 PM, 3/10/2023 11:41:11 AM, 3/15/2023 5:33:41 AM, 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -8.904010210387916 (T) = (0 -16.057294000553572) / Math.Sqrt((0.7891327362532914 / (16)) + (1.1277729601197295 / (33))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (33) - 2, .025) and -0.1908030813085121 = (13.484424295332726 - 16.057294000553572) / 13.484424295332726 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

System.Tests.Perf_Int32.ParseHex(value: "7FFFFFFF")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 19.02748244608215 > 16.203691373234975.
IsChangePoint: Marked as a change because one of 2/14/2023 11:21:41 AM, 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -14.016206848187618 (T) = (0 -19.489907440728587) / Math.Sqrt((0.8436169260970453 / (16)) + (0.8907590014736675 / (33))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (33) - 2, .025) and -0.254782377327061 = (15.532500131414032 - 19.489907440728587) / 15.532500131414032 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

System.Tests.Perf_Int32.ParseSpan(value: "2147483647")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 23.57805118163993 > 22.34321693431005.
IsChangePoint: Marked as a change because one of 2/16/2023 4:50:02 PM, 3/10/2023 12:53:36 AM, 3/15/2023 5:33:41 AM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -3.2093549220671167 (T) = (0 -24.22186774951004) / Math.Sqrt((4.0506733803809984 / (12)) + (3.4988943522717144 / (37))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (37) - 2, .025) and -0.0954089230000209 = (22.112169474731928 - 24.22186774951004) / 22.112169474731928 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7dbf6a5f2fd636db350f9cc31b2dfd3af0eff1a8
Compare eb71e484d2d4a7ae002ea524d2edead773289bbb
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in PerfLabTests.CastingPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ObjObjrefValueType - Duration of single invocation 156.43 μs 227.73 μs 1.46 0.05 False Trace Trace
ObjInt - Duration of single invocation 155.92 μs 228.25 μs 1.46 0.04 False Trace Trace

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'PerfLabTests.CastingPerf*'

Payloads

Baseline
Compare

Histogram

PerfLabTests.CastingPerf.ObjObjrefValueType

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 227.73487922705317 > 163.77269067141088.
IsChangePoint: Marked as a change because one of 2/20/2023 10:31:48 AM, 3/6/2023 11:14:52 AM, 3/20/2023 8:31:51 AM, 3/27/2023 4:20:55 AM, 3/31/2023 8:05:26 PM, 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -14.936730671673677 (T) = (0 -226501.5766717023) / Math.Sqrt((310726813.29065925 / (16)) + (3064001.183003511 / (33))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (33) - 2, .025) and -0.411046189198672 = (160520.3135131471 - 226501.5766717023) / 160520.3135131471 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

PerfLabTests.CastingPerf.ObjInt

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 228.2457822712418 > 163.71869090346536.
IsChangePoint: Marked as a change because one of 2/20/2023 10:31:48 AM, 3/6/2023 11:14:52 AM, 3/20/2023 8:31:51 AM, 3/27/2023 4:20:55 AM, 3/31/2023 8:05:26 PM, 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -15.459119136361023 (T) = (0 -226350.98089952642) / Math.Sqrt((289883804.0014927 / (16)) + (4621012.597156613 / (33))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (33) - 2, .025) and -0.41208532674460674 = (160295.54065358886 - 226350.98089952642) / 160295.54065358886 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7dbf6a5f2fd636db350f9cc31b2dfd3af0eff1a8
Compare eb71e484d2d4a7ae002ea524d2edead773289bbb
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Byte

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Parse - Duration of single invocation 13.56 ns 15.22 ns 1.12 0.51 False Trace Trace

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Byte*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Byte.Parse(value: "255")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.224337424197927 > 12.24546245891612.
IsChangePoint: Marked as a change because one of 2/16/2023 4:50:02 PM, 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -15.092310465091057 (T) = (0 -15.382216161735908) / Math.Sqrt((0.6686105833535426 / (16)) + (0.31790130192750504 / (33))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (33) - 2, .025) and -0.28615610786426127 = (11.95983603209644 - 15.382216161735908) / 11.95983603209644 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Baseline
Compare
Diff

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@kunalspathak
Copy link
Member

dotnet/runtime#84587

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kunalspathak kunalspathak

Labels
arch-x86 branch-refs/heads/main kind-micro os-windows perf-regression runtime-coreclr untriaged
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@kunalspathak