Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

9 Regressions in PerfLabTests.GetMember #93741

Closed
performanceautofiler bot opened this issue Oct 19, 2023 · 4 comments · Fixed by #95872
Closed

9 Regressions in PerfLabTests.GetMember #93741

performanceautofiler bot opened this issue Oct 19, 2023 · 4 comments · Fixed by #95872
Assignees
@stephentoub
Labels
area-System.Reflection runtime-coreclr specific to the CoreCLR runtime tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Milestone
9.0.0

Comments

@performanceautofiler
Copy link

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 613e4fecde90a05d1df0cda69392434514ac18f5
Compare 0a77fd44c350894f0ea84a46f5188b9073214d26
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in PerfLabTests.GetMember

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
221.68 μs 278.57 μs 1.26 0.02 False
73.98 μs 95.70 μs 1.29 0.02 False
371.25 μs 456.61 μs 1.23 0.02 False
148.61 μs 180.35 μs 1.21 0.02 False
297.59 μs 371.52 μs 1.25 0.03 False
1.11 ms 1.41 ms 1.27 0.07 False
741.17 μs 911.39 μs 1.23 0.05 False
910.50 μs 1.16 ms 1.27 0.03 False
1.49 ms 1.93 ms 1.30 0.05 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'PerfLabTests.GetMember*'

Payloads

Baseline
Compare

PerfLabTests.GetMember.GetMethod3

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 5/31/2023 1:59:31 PM, 6/5/2023 10:43:22 AM, 10/10/2023 10:07:15 PM, 10/18/2023 4:42:02 PM falls between 10/9/2023 8:09:24 PM and 10/18/2023 4:42:02 PM.
IsRegressionStdDev: Marked as regression because -32.3944600633967 (T) = (0 -275072.22820271866) / Math.Sqrt((889674.8230223494 / (6)) + (22971229.956829682 / (9))) is less than -2.160368656462815 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (9) - 2, .025) and -0.23997782144653412 = (221836.40984951225 - 275072.22820271866) / 221836.40984951225 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

PerfLabTests.GetMember.GetMethod1

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 5/31/2023 1:59:31 PM, 6/5/2023 10:43:22 AM, 10/10/2023 10:07:15 PM, 10/18/2023 4:42:02 PM falls between 10/9/2023 8:09:24 PM and 10/18/2023 4:42:02 PM.
IsRegressionStdDev: Marked as regression because -26.892601057787957 (T) = (0 -93602.55159970208) / Math.Sqrt((110743.80163713293 / (6)) + (4525014.276401095 / (9))) is less than -2.160368656462815 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (9) - 2, .025) and -0.26171138964860147 = (74186.97522083182 - 93602.55159970208) / 74186.97522083182 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

PerfLabTests.GetMember.GetMethod5

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 5/31/2023 1:59:31 PM, 6/5/2023 10:43:22 AM, 10/10/2023 10:07:15 PM, 10/18/2023 4:42:02 PM falls between 10/9/2023 8:09:24 PM and 10/18/2023 4:42:02 PM.
IsRegressionStdDev: Marked as regression because -30.19655409469621 (T) = (0 -466109.2520683187) / Math.Sqrt((1329211.0561002628 / (6)) + (88841336.37915991 / (9))) is less than -2.160368656462815 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (9) - 2, .025) and -0.25915123235708637 = (370177.33858368924 - 466109.2520683187) / 370177.33858368924 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

PerfLabTests.GetMember.GetMethod2

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 5/31/2023 1:59:31 PM, 6/5/2023 10:43:22 AM, 10/10/2023 10:07:15 PM, 10/18/2023 4:42:02 PM falls between 10/9/2023 8:09:24 PM and 10/18/2023 4:42:02 PM.
IsRegressionStdDev: Marked as regression because -25.315003329250736 (T) = (0 -186328.03501914858) / Math.Sqrt((961369.6579639735 / (6)) + (19769628.043357894 / (9))) is less than -2.160368656462815 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (9) - 2, .025) and -0.2635465502147761 = (147464.32174380656 - 186328.03501914858) / 147464.32174380656 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

PerfLabTests.GetMember.GetMethod4

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 5/31/2023 1:59:31 PM, 6/5/2023 10:43:22 AM, 10/10/2023 10:07:15 PM, 10/18/2023 4:42:02 PM falls between 10/9/2023 8:09:24 PM and 10/18/2023 4:42:02 PM.
IsRegressionStdDev: Marked as regression because -6.900356682142859 (T) = (0 -383883.6290003003) / Math.Sqrt((320412351.4856488 / (6)) + (674513790.7360713 / (9))) is less than -2.160368656462815 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (9) - 2, .025) and -0.25571606400524244 = (305708.9416980634 - 383883.6290003003) / 305708.9416980634 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

PerfLabTests.GetMember.GetMethod15

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 8/2/2023 4:26:52 PM, 10/10/2023 10:07:15 PM, 10/18/2023 4:42:02 PM falls between 10/9/2023 8:09:24 PM and 10/18/2023 4:42:02 PM.
IsRegressionStdDev: Marked as regression because -22.385856498302267 (T) = (0 -1407118.8837774424) / Math.Sqrt((153367357.76972824 / (6)) + (1213308946.8109193 / (9))) is less than -2.160368656462815 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (9) - 2, .025) and -0.2523000943250837 = (1123627.5475454603 - 1407118.8837774424) / 1123627.5475454603 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

PerfLabTests.GetMember.GetMethod10

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 4/27/2023 10:53:57 PM, 10/10/2023 10:07:15 PM, 10/18/2023 4:42:02 PM falls between 10/9/2023 8:09:24 PM and 10/18/2023 4:42:02 PM.
IsRegressionStdDev: Marked as regression because -18.636747012425275 (T) = (0 -931731.028136682) / Math.Sqrt((111068.23467145863 / (6)) + (938244357.5098697 / (9))) is less than -2.160368656462815 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (9) - 2, .025) and -0.2566707198401307 = (741428.135013135 - 931731.028136682) / 741428.135013135 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

PerfLabTests.GetMember.GetMethod12

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 4/29/2023 4:23:29 AM, 10/10/2023 10:07:15 PM, 10/18/2023 4:42:02 PM falls between 10/9/2023 8:09:24 PM and 10/18/2023 4:42:02 PM.
IsRegressionStdDev: Marked as regression because -21.054602763856753 (T) = (0 -1133688.2765882201) / Math.Sqrt((86780721.66693616 / (6)) + (1018901935.0409738 / (9))) is less than -2.160368656462815 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (9) - 2, .025) and -0.2655801481893287 = (895785.4452838828 - 1133688.2765882201) / 895785.4452838828 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

PerfLabTests.GetMember.GetMethod20

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 4/28/2023 7:22:08 PM, 10/10/2023 10:07:15 PM, 10/18/2023 4:42:02 PM falls between 10/9/2023 8:09:24 PM and 10/18/2023 4:42:02 PM.
IsRegressionStdDev: Marked as regression because -17.18560559768971 (T) = (0 -1868932.459490741) / Math.Sqrt((257134052.9352805 / (6)) + (3246474447.145263 / (9))) is less than -2.160368656462815 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (9) - 2, .025) and -0.22658492725497287 = (1523687.775679182 - 1868932.459490741) / 1523687.775679182 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler performanceautofiler bot added arch-arm64 os-windows runtime-coreclr specific to the CoreCLR runtime untriaged New issue has not been triaged by the area owner labels Oct 19, 2023
@performanceautofiler performanceautofiler bot mentioned this issue Oct 19, 2023
Closed
@EgorBo EgorBo changed the title [Perf] Windows/arm64: 9 Regressions on 10/11/2023 7:15:38 PM 9 Regressions in PerfLabTests.GetMember Oct 19, 2023
@EgorBo EgorBo transferred this issue from dotnet/perf-autofiling-issues Oct 19, 2023
@dotnet-issue-labeler dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Oct 19, 2023
@EgorBo
Copy link
Member

EgorBo commented Oct 19, 2023

Same on Ampere-arm64-windows:

@EgorBo
Copy link
Member

EgorBo commented Oct 19, 2023

We think it's caused by #93115 cc @stephentoub
image

@EgorBo EgorBo added area-System.Reflection tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark and removed arch-arm64 os-windows needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners labels Oct 19, 2023
@ghost
Copy link

ghost commented Oct 19, 2023

Tagging subscribers to this area: @dotnet/area-system-reflection
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline 613e4fecde90a05d1df0cda69392434514ac18f5
Compare 0a77fd44c350894f0ea84a46f5188b9073214d26
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in PerfLabTests.GetMember

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
221.68 μs 278.57 μs 1.26 0.02 False
73.98 μs 95.70 μs 1.29 0.02 False
371.25 μs 456.61 μs 1.23 0.02 False
148.61 μs 180.35 μs 1.21 0.02 False
297.59 μs 371.52 μs 1.25 0.03 False
1.11 ms 1.41 ms 1.27 0.07 False
741.17 μs 911.39 μs 1.23 0.05 False
910.50 μs 1.16 ms 1.27 0.03 False
1.49 ms 1.93 ms 1.30 0.05 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'PerfLabTests.GetMember*'

Payloads

Baseline
Compare

PerfLabTests.GetMember.GetMethod3

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 5/31/2023 1:59:31 PM, 6/5/2023 10:43:22 AM, 10/10/2023 10:07:15 PM, 10/18/2023 4:42:02 PM falls between 10/9/2023 8:09:24 PM and 10/18/2023 4:42:02 PM.
IsRegressionStdDev: Marked as regression because -32.3944600633967 (T) = (0 -275072.22820271866) / Math.Sqrt((889674.8230223494 / (6)) + (22971229.956829682 / (9))) is less than -2.160368656462815 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (9) - 2, .025) and -0.23997782144653412 = (221836.40984951225 - 275072.22820271866) / 221836.40984951225 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

PerfLabTests.GetMember.GetMethod1

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 5/31/2023 1:59:31 PM, 6/5/2023 10:43:22 AM, 10/10/2023 10:07:15 PM, 10/18/2023 4:42:02 PM falls between 10/9/2023 8:09:24 PM and 10/18/2023 4:42:02 PM.
IsRegressionStdDev: Marked as regression because -26.892601057787957 (T) = (0 -93602.55159970208) / Math.Sqrt((110743.80163713293 / (6)) + (4525014.276401095 / (9))) is less than -2.160368656462815 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (9) - 2, .025) and -0.26171138964860147 = (74186.97522083182 - 93602.55159970208) / 74186.97522083182 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

PerfLabTests.GetMember.GetMethod5

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 5/31/2023 1:59:31 PM, 6/5/2023 10:43:22 AM, 10/10/2023 10:07:15 PM, 10/18/2023 4:42:02 PM falls between 10/9/2023 8:09:24 PM and 10/18/2023 4:42:02 PM.
IsRegressionStdDev: Marked as regression because -30.19655409469621 (T) = (0 -466109.2520683187) / Math.Sqrt((1329211.0561002628 / (6)) + (88841336.37915991 / (9))) is less than -2.160368656462815 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (9) - 2, .025) and -0.25915123235708637 = (370177.33858368924 - 466109.2520683187) / 370177.33858368924 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

PerfLabTests.GetMember.GetMethod2

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 5/31/2023 1:59:31 PM, 6/5/2023 10:43:22 AM, 10/10/2023 10:07:15 PM, 10/18/2023 4:42:02 PM falls between 10/9/2023 8:09:24 PM and 10/18/2023 4:42:02 PM.
IsRegressionStdDev: Marked as regression because -25.315003329250736 (T) = (0 -186328.03501914858) / Math.Sqrt((961369.6579639735 / (6)) + (19769628.043357894 / (9))) is less than -2.160368656462815 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (9) - 2, .025) and -0.2635465502147761 = (147464.32174380656 - 186328.03501914858) / 147464.32174380656 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

PerfLabTests.GetMember.GetMethod4

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 5/31/2023 1:59:31 PM, 6/5/2023 10:43:22 AM, 10/10/2023 10:07:15 PM, 10/18/2023 4:42:02 PM falls between 10/9/2023 8:09:24 PM and 10/18/2023 4:42:02 PM.
IsRegressionStdDev: Marked as regression because -6.900356682142859 (T) = (0 -383883.6290003003) / Math.Sqrt((320412351.4856488 / (6)) + (674513790.7360713 / (9))) is less than -2.160368656462815 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (9) - 2, .025) and -0.25571606400524244 = (305708.9416980634 - 383883.6290003003) / 305708.9416980634 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

PerfLabTests.GetMember.GetMethod15

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 8/2/2023 4:26:52 PM, 10/10/2023 10:07:15 PM, 10/18/2023 4:42:02 PM falls between 10/9/2023 8:09:24 PM and 10/18/2023 4:42:02 PM.
IsRegressionStdDev: Marked as regression because -22.385856498302267 (T) = (0 -1407118.8837774424) / Math.Sqrt((153367357.76972824 / (6)) + (1213308946.8109193 / (9))) is less than -2.160368656462815 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (9) - 2, .025) and -0.2523000943250837 = (1123627.5475454603 - 1407118.8837774424) / 1123627.5475454603 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

PerfLabTests.GetMember.GetMethod10

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 4/27/2023 10:53:57 PM, 10/10/2023 10:07:15 PM, 10/18/2023 4:42:02 PM falls between 10/9/2023 8:09:24 PM and 10/18/2023 4:42:02 PM.
IsRegressionStdDev: Marked as regression because -18.636747012425275 (T) = (0 -931731.028136682) / Math.Sqrt((111068.23467145863 / (6)) + (938244357.5098697 / (9))) is less than -2.160368656462815 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (9) - 2, .025) and -0.2566707198401307 = (741428.135013135 - 931731.028136682) / 741428.135013135 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

PerfLabTests.GetMember.GetMethod12

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 4/29/2023 4:23:29 AM, 10/10/2023 10:07:15 PM, 10/18/2023 4:42:02 PM falls between 10/9/2023 8:09:24 PM and 10/18/2023 4:42:02 PM.
IsRegressionStdDev: Marked as regression because -21.054602763856753 (T) = (0 -1133688.2765882201) / Math.Sqrt((86780721.66693616 / (6)) + (1018901935.0409738 / (9))) is less than -2.160368656462815 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (9) - 2, .025) and -0.2655801481893287 = (895785.4452838828 - 1133688.2765882201) / 895785.4452838828 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

PerfLabTests.GetMember.GetMethod20

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 4/28/2023 7:22:08 PM, 10/10/2023 10:07:15 PM, 10/18/2023 4:42:02 PM falls between 10/9/2023 8:09:24 PM and 10/18/2023 4:42:02 PM.
IsRegressionStdDev: Marked as regression because -17.18560559768971 (T) = (0 -1868932.459490741) / Math.Sqrt((257134052.9352805 / (6)) + (3246474447.145263 / (9))) is less than -2.160368656462815 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (9) - 2, .025) and -0.22658492725497287 = (1523687.775679182 - 1868932.459490741) / 1523687.775679182 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: -
Labels:

area-System.Reflection, tenet-performance, tenet-performance-benchmarks, untriaged, runtime-coreclr
Milestone: -

@EgorBo
Copy link
Member

EgorBo commented Oct 19, 2023

Also, some in dotnet/perf-autofiling-issues#23290

@stephentoub stephentoub self-assigned this Oct 20, 2023
@buyaa-n buyaa-n added this to the 9.0.0 milestone Oct 25, 2023
@ghost ghost removed the untriaged New issue has not been triaged by the area owner label Oct 25, 2023
@stephentoub stephentoub mentioned this issue Dec 11, 2023
Merged
@ghost ghost added the in-pr There is an active PR which will close this issue when it is merged label Dec 11, 2023
@ghost ghost removed the in-pr There is an active PR which will close this issue when it is merged label Dec 12, 2023
@github-actions github-actions bot locked and limited conversation to collaborators Jan 11, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@stephentoub stephentoub

Labels
area-System.Reflection runtime-coreclr specific to the CoreCLR runtime tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Projects
None yet
Milestone
9.0.0
Development

Successfully merging a pull request may close this issue.

Revert "Avoid a few array allocations with DefaultBinder (#93115)" stephentoub/runtime
3 participants
@EgorBo @stephentoub @buyaa-n