Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Start-up performance degradation of.NET 8 in Amazon Linux 2023 over Amazon Linux 2 #96740

Closed
birojnayak opened this issue Jan 10, 2024 · 8 comments
Closed

Start-up performance degradation of.NET 8 in Amazon Linux 2023 over Amazon Linux 2 #96740

birojnayak opened this issue Jan 10, 2024 · 8 comments
Labels
area-System.Security tenet-performance Performance related issue

Comments

@birojnayak
Copy link
Contributor

birojnayak commented Jan 10, 2024 •
edited
Loading

Description

We are observing a start-up performance degradation of.NET 8 in Amazon Linux 2023 over Amazon Linux 2 for https calls. The performance is intact if it's a local API call (file read/write).

Configuration

  • Which version of .NET is the code running on?
    • .NET 8
  • What OS version, and what distro if applicable?
    • Amazon Linux 2023
  • What is the architecture (x64, x86, ARM, ARM64)?
    *x86_64 , ARM64
  • If relevant, what are the specs of the machine?

Regression?

  • Is this a regression from a previous build or release of .NET Core, or from .NET Framework?
    • I have not tried the performance between.NET 6 and.NET 8 in Amazon Linux 2023, but if needed, I can get that information.

Data

Amazon Linux 2

// AfterActualRun
WorkloadResult 1: 1 op, 90633901.00 ns, 90.6339 ms/op <= ColdStart
WorkloadResult 2: 1 op, 1716177.00 ns, 1.7162 ms/op
WorkloadResult 3: 1 op, 1006949.00 ns, 1.0069 ms/op
WorkloadResult 4: 1 op, 894721.00 ns, 894.7210 us/op
WorkloadResult 5: 1 op, 997069.00 ns, 997.0690 us/op

// * Summary *
BenchmarkDotNet v0.13.12, Amazon Linux 2
Intel Xeon Platinum 8275CL CPU 3.00GHz, 1 CPU, 2 logical cores and 1 physical core
.NET SDK 8.0.101
[Host] : .NET 8.0.1 (8.0.123.58001), X64 RyuJIT AVX-512F+CD+BW+DQ+VL
Job-LDEQOF : .NET 8.0.1 (8.0.123.58001), X64 RyuJIT AVX-512F+CD+BW+DQ+VL

IterationCount=5 RunStrategy=ColdStart

Method Mean Error StdDev Median Min Max
Foo 19.95 ms 161.52 ms 41.95 ms 1.330 ms 0.8515 ms 94.98 ms

Amazon Linux 2023

// AfterActualRun
WorkloadResult 1: 1 op, 137960338.00 ns, 137.9603 ms/op <= ColdStart
WorkloadResult 2: 1 op, 2505819.00 ns, 2.5058 ms/op
WorkloadResult 3: 1 op, 1495447.00 ns, 1.4954 ms/op
WorkloadResult 4: 1 op, 1204715.00 ns, 1.2047 ms/op
WorkloadResult 5: 1 op, 1379885.00 ns, 1.3799 ms/op

// * Summary *

BenchmarkDotNet v0.13.12, Amazon Linux 2023
Intel Xeon Platinum 8275CL CPU 3.00GHz, 1 CPU, 2 logical cores and 1 physical core
.NET SDK 8.0.100
[Host] : .NET 8.0.0 (8.0.23.53103), X64 RyuJIT AVX-512F+CD+BW+DQ+VL
Job-GQCBXH : .NET 8.0.0 (8.0.23.53103), X64 RyuJIT AVX-512F+CD+BW+DQ+VL

IterationCount=5 RunStrategy=ColdStart

Method Mean Error StdDev Median Min Max
Foo 28.91 ms 234.75 ms 60.96 ms 1.495 ms 1.205 ms 138.0 ms

Analysis

  • Using dotnet-trace + perfview, we were able to pin point the problem in the Cryptography library (more details in the PerfView regression result info.zip) (thanks to @mconnew and @adamsitnik for helping with initial analysis). It seems (not sure) that the problem is with loading certificates from the local store. We would like to confirm ?  and know the reason behind it. Along with that, what's the right way to resolve it?
  • In the attachment I have added the sample app , the benchmark test and Perfview regression screenshot (base of the prefview regression is Amazon Linux 2). If you need full trace , let me know I can provide.
@birojnayak birojnayak added the tenet-performance Performance related issue label Jan 10, 2024
@dotnet-issue-labeler dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Jan 10, 2024
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Jan 10, 2024
@birojnayak birojnayak changed the title Start-up performance degradation of.NET 8 in Amazon Linux 3 over Amazon Linux 2 Start-up performance degradation of.NET 8 in Amazon Linux 2023 over Amazon Linux 2 Jan 10, 2024
@adamsitnik adamsitnik added area-System.Security and removed needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners labels Jan 10, 2024
@ghost
Copy link

ghost commented Jan 10, 2024

Tagging subscribers to this area: @dotnet/area-system-security, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

Description

We are observing a start-up performance degradation of.NET 8 in Amazon Linux 2023 over Amazon Linux 2 for https calls. The performance is intact if it's a local API call (file read/write).

Configuration

  • Which version of .NET is the code running on?
    • .NET 8
  • What OS version, and what distro if applicable?
    • Amazon Linux 2023
  • What is the architecture (x64, x86, ARM, ARM64)?
    *x86_64
  • If relevant, what are the specs of the machine?

Regression?

  • Is this a regression from a previous build or release of .NET Core, or from .NET Framework?
    • I have not tried the performance between.NET 6 and.NET 8 in Amazon Linux 2023, but if needed, I can get that information.

Data

Amazon Linux 2

// AfterActualRun
WorkloadResult 1: 1 op, 90633901.00 ns, 90.6339 ms/op <= ColdStart
WorkloadResult 2: 1 op, 1716177.00 ns, 1.7162 ms/op
WorkloadResult 3: 1 op, 1006949.00 ns, 1.0069 ms/op
WorkloadResult 4: 1 op, 894721.00 ns, 894.7210 us/op
WorkloadResult 5: 1 op, 997069.00 ns, 997.0690 us/op

// * Summary *
BenchmarkDotNet v0.13.12, Amazon Linux 2
Intel Xeon Platinum 8275CL CPU 3.00GHz, 1 CPU, 2 logical cores and 1 physical core
.NET SDK 8.0.101
[Host] : .NET 8.0.1 (8.0.123.58001), X64 RyuJIT AVX-512F+CD+BW+DQ+VL
Job-LDEQOF : .NET 8.0.1 (8.0.123.58001), X64 RyuJIT AVX-512F+CD+BW+DQ+VL

IterationCount=5 RunStrategy=ColdStart

Method Mean Error StdDev Median Min Max
Foo 19.95 ms 161.52 ms 41.95 ms 1.330 ms 0.8515 ms 94.98 ms

Amazon Linux 2023

// AfterActualRun
WorkloadResult 1: 1 op, 137960338.00 ns, 137.9603 ms/op <= ColdStart
WorkloadResult 2: 1 op, 2505819.00 ns, 2.5058 ms/op
WorkloadResult 3: 1 op, 1495447.00 ns, 1.4954 ms/op
WorkloadResult 4: 1 op, 1204715.00 ns, 1.2047 ms/op
WorkloadResult 5: 1 op, 1379885.00 ns, 1.3799 ms/op

// * Summary *

BenchmarkDotNet v0.13.12, Amazon Linux 2023
Intel Xeon Platinum 8275CL CPU 3.00GHz, 1 CPU, 2 logical cores and 1 physical core
.NET SDK 8.0.100
[Host] : .NET 8.0.0 (8.0.23.53103), X64 RyuJIT AVX-512F+CD+BW+DQ+VL
Job-GQCBXH : .NET 8.0.0 (8.0.23.53103), X64 RyuJIT AVX-512F+CD+BW+DQ+VL

IterationCount=5 RunStrategy=ColdStart

Method Mean Error StdDev Median Min Max
Foo 28.91 ms 234.75 ms 60.96 ms 1.495 ms 1.205 ms 138.0 ms

Analysis

  • Using dotnet-trace + perfview, we were able to pin point the problem in the Cryptography library (more details in the PerfView regression result info.zip) (thanks to @mconnew and @adamsitnik for helping with initial analysis). It seems (not sure) that the problem is with loading certificates from the local store. We would like to confirm ?  and know the reason behind it. Along with that, what's the right way to resolve it?
  • In the attachment I have added the sample app , the benchmark test and Perfview regression screenshot (base of the prefview regression is Amazon Linux 2). If you need full trace , let me know I can provide.
Author: birojnayak
Assignees: -
Labels:

area-System.Security, tenet-performance, untriaged, needs-area-label
Milestone: -

@adamsitnik
Copy link
Member

The PerfView diff that shows where most of the diff comes from:

image

@birojnayak
Copy link
Contributor Author

birojnayak commented Jan 10, 2024 •
edited
Loading

@adamsitnik thanks.. Along with above screenshot (shared by Adam), sample app is also attached in the info.zip in the Analysis section.

@birojnayak
Copy link
Contributor Author

birojnayak commented Jan 12, 2024 •
edited
Loading

Consulted with @jkotas .. we are assuming to be OpenSSL 3.0 issue , which AL2023 uses. Let us know if there is a known workaround in runtime.

FYI.. I also bench marked the LoadMachineStores method. We see performance degrade between AL2 and AL2023 with cert file and cert directory same across (OS) with same size.

@vcsjones
Copy link
Member

vcsjones commented Jan 13, 2024 •
edited
Loading

Sounds like this is openssl/openssl#16871. It is known that OpenSSL 3's PEM_read_bio_X509 is slower than OpenSSL 1.1. This function is used to perform the initial read of the root certificates.

@birojnayak
Copy link
Contributor Author

@vcsjones thank you.. is there a possibility before reading file, we can check if the file is already read. I understand we are honoring the uniqueness in the certs, but as you see if same file is read multiple times, we are causing performance issues too. Let me know if you agree, happy to submit a PR.

@birojnayak birojnayak mentioned this issue Jan 21, 2024
Merged
@vcsjones
Copy link
Member

Between #97267 and #97827 it seems like start up time is improved. The OpenSSL API remains a tad slow, but the performance was "won" back by other means.

Should we close this issue out then?

@birojnayak
Copy link
Contributor Author

closing the issue as fix has been deployed.

@ghost ghost removed the untriaged New issue has not been triaged by the area owner label Feb 19, 2024
@github-actions github-actions bot locked and limited conversation to collaborators Mar 21, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
area-System.Security tenet-performance Performance related issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@vcsjones @adamsitnik @birojnayak