[Perf] Linux/x64: 8 Regressions on 5/12/2023 7:53:45 AM

[performanceautofiler]

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	b92c1d23c1ad75d72f947c3e7fff5e9b73bd9cb5
Compare	7eaf4b6d4dd2e67387bca7e5fb29e2db47f2589b
Diff	Diff
Configs	CompilationMode:wasm, RunKind:blazor_scenarios

Regressions in SOD - Minimum Blazor Template - Publish

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
SOD - Minimum Blazor Template - Publish - Aggregate - .js	405.10 KB	83.98 KB	0.21	0.00	True
SOD - Minimum Blazor Template - Publish - pub/wwwroot/_framework/dotnet.VERSION.js	350.53 KB	29.40 KB	0.08	0.00	True
SOD - Minimum Blazor Template - Publish - Aggregate - .json	3.96 KB	3.83 KB	0.97	0.00	True
SOD - Minimum Blazor Template - Publish - pub/wwwroot/_framework/blazor.boot.json	3.52 KB	3.39 KB	0.96	0.00	True
SOD - Minimum Blazor Template - Publish - Total Uncompressed _framework	3.91 MB	5.04 MB	1.29	0.00	True
SOD - Minimum Blazor Template - Publish - pub	4.66 MB	5.79 MB	1.24	0.00	True
SOD - Minimum Blazor Template - Publish - SOD - Minimum Blazor Template - Publish	4.66 MB	5.79 MB	1.24	0.00	True
SOD - Minimum Blazor Template - Publish - Aggregate - .wasm	1.29 MB	2.74 MB	2.12	0.00	True

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'SOD - Minimum Blazor Template - Publish*'

Payloads

Baseline
Compare

Histogram

SOD - Minimum Blazor Template - Publish

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 1% greater than the baseline, and the value was not too small.
IsRegressionBase: Marked as regression because the compare was 1% greater than the baseline, and the value was not too small.
IsRegressionBase: Marked as regression because the compare was 1% greater than the baseline, and the value was not too small.
IsRegressionBase: Marked as regression because the compare was 1% greater than the baseline, and the value was not too small.
IsImprovementBase: Marked as improvement because the compare was 1% less than the baseline, and the value was not too small.
IsRegressionBase: Marked as regression because the compare was 1% greater than the baseline, and the value was not too small.
IsRegressionBase: Marked as regression because the compare was 1% greater than the baseline, and the value was not too small.
IsRegressionBase: Marked as regression because the compare was 1% greater than the baseline, and the value was not too small.
IsRegressionBase: Marked as regression because the compare was 1% greater than the baseline, and the value was not too small.
IsImprovementBase: Marked as improvement because the compare was 1% less than the baseline, and the value was not too small.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[radekdoulik]

@pavelsavara looks like dotnet/runtime#85730

[pavelsavara]

It seems to me that the test itself needs fixing, where do I find the code which measures this ?

[radekdoulik]

These blazor scenarios are located here https://github.com/dotnet/performance/blob/main/eng/performance/blazor_scenarios.proj and the blazor* folders here https://github.com/dotnet/performance/tree/main/src/scenarios

[radekdoulik]

@LoopedBard3 how could we see more details about this regression? is there an archive with the build somewhere or the log? are these clean builds?

[LoopedBard3]

@radekdoulik Since this is wasm, this blazor scenario set is defined here: https://github.com/dotnet/runtime/blob/main/eng/testing/performance/blazor_perf.proj, although it appears to be the same as the blazor_scenarios.proj file in the performance repo.

When it comes to the archive and logs, if you find the runtime git commits you want to view, the runs are completed in this pipeline: https://dev.azure.com/dnceng/internal/_build?definitionId=702&_a=summary. I can find the two specific pipeline runs if you can provide the baseline and compare commits you want. Once in a pipeline run, the job for the test will be "Performance linux x64 release wasm JIT blazor_scenarios perftiger NoJS False net8.0" and artifacts can be found in the Azdo artifacts or through the helix API by clicking the Send to Helix job step and following the work item links.

[pavelsavara]

Where is the code that chooses which files should be measured ?

[LoopedBard3]

Do you mean like which csproj files are used for the tests? If so, since this was a wasm compilation and blazor scenarios, the run was specified here: https://github.com/dotnet/runtime/blob/main/eng/pipelines/coreclr/perf-wasm-jobs.yml#L153-L173

[pavelsavara]

No I mean, something found file dotnet.wasm in the past and reported it's size. Now the file was renamed to dotnet.native.wasm and it's reported twice as big. I want to have look at the code which finds the file and it's size.

[LoopedBard3]

Ah, I think I understand. The primary code for the size gathering tool is located https://github.com/dotnet/performance/blob/main/src/tools/ScenarioMeasurement/SizeOnDisk/SizeOnDisk.cs.

The general flow is the app is published to a "pub" directory, the SizeOnDisk test is started with https://github.com/dotnet/performance/blob/main/src/scenarios/blazorminapp/test.py, the test.py calls https://github.com/dotnet/performance/blob/main/src/scenarios/shared/runner.py with L783-L790 using a wrapper to call the SizeOnDisk tool.

[pavelsavara]

cc @SamMonoRT

[pavelsavara]

Probably caused by dotnet/runtime#88142 and fixed by dotnet/runtime#89079

[pavelsavara]

@lewing could you please teach me how to validate if this is fixed or not ? Many thanks!

[lewing]

I think we're looking into all the size stuff now and there isn't anything specific worth tracking here so we can call this closed, @radekdoulik feel free to close it.