Add Azure 3.0 helix test #106330

richlander · 2024-08-13T14:25:46Z

No description provided.

richlander · 2024-08-13T18:50:33Z

Is this change even getting triggered by this PR run? I'm guessing not? I wasn't able to find it in the build logs. I looked within Libraries Test Run release coreclr linux x64 Debug.

It is the same place as the existing Mariner image.

runtime/eng/pipelines/libraries/helix-queues-setup.yml

Lines 64 to 70 in 8e0927f

	- ${{ if or(ne(parameters.jobParameters.testScope, 'outerloop'), ne(parameters.jobParameters.runtimeFlavor, 'mono')) }}:
	- ${{ if or(eq(parameters.jobParameters.isExtraPlatformsBuild, true), eq(parameters.jobParameters.includeAllPlatforms, true)) }}:
	- SLES.15.Amd64.Open
	- (Fedora.38.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:fedora-38-helix
	- Ubuntu.2204.Amd64.Open
	- (Debian.11.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:debian-11-helix-amd64
	- (Mariner.2.0.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:cbl-mariner-2.0-helix-amd64

If not, how do I trigger the outer loop? Is this the right place for Azure Linux validation going forward?

jkotas · 2024-08-13T19:39:29Z

/azp help

azure-pipelines · 2024-08-13T19:39:34Z

Supported commands help: Get descriptions, examples and documentation about supported commands Example: help "command_name" list: List all pipelines for this repository using a comment. Example: "list" run: Run all pipelines or specific pipelines for this repository using a comment. Use this command by itself to trigger all related pipelines, or specify specific pipelines to run. Example: "run" or "run pipeline_name, pipeline_name, pipeline_name" where: Report back the Azure DevOps orgs that are related to this repository and org Example: "where" See additional documentation.

jkotas · 2024-08-13T19:40:11Z

/azp list

azure-pipelines · 2024-08-13T19:40:17Z

CI/CD Pipelines for this repository: runtime-coreclr outerloop runtime-coreclr jitstress runtime-coreclr jitstressregs runtime-coreclr jitstress2-jitstressregs runtime-coreclr gcstress0x3-gcstress0xc runtime-coreclr gcstress-extra runtime-coreclr r2r-extra runtime-coreclr jitstress-isas-x86 runtime-coreclr jitstress-isas-arm runtime-coreclr jitstressregs-x86 runtime-coreclr libraries-jitstressregs runtime-coreclr libraries-jitstress2-jitstressregs runtime-coreclr r2r runtime-coreclr runincontext runtime-coreclr crossgen2 runtime-libraries-coreclr outerloop runtime-libraries-coreclr outerloop-windows runtime-libraries-coreclr outerloop-linux runtime-libraries-coreclr outerloop-osx runtime runtime-libraries enterprise-linux runtime-libraries stress-http runtime-libraries stress-ssl runtime-dev-innerloop runtime-coreclr crossgen2 outerloop coreclr-release-outerloop-nightly runtime-coreclr crossgen2-composite runtime-jit-experimental runtime-coreclr libraries-jitstress dotnet-linker-tests runtime-coreclr ilasm runtime-coreclr crossgen2-composite gcstress runtime-staging runtime-coreclr pgo runtime-coreclr libraries-pgo Antigen runtime-community Fuzzlyn runtime-coreclr superpmi-replay runtime-wasm runtime-coreclr superpmi-diffs runtime-coreclr superpmi-asmdiffs-checked-release runtime-extra-platforms jit-cfg runtime-wasm-perf runtime-llvm runtime-coreclr jitstress-random runtime-coreclr libraries-jitstress-random runtime-android-grpc-client-tests runtime-wasm-non-libtests runtime-android runtime-androidemulator runtime-ioslike runtime-ioslikesimulator runtime-linuxbionic runtime-maccatalyst runtime-coreclr pgostress runtime-coreclr jitstress-isas-avx512 runtime-libraries-mono outerloop runtime-sanitized runtime-wasm-dbgtests runtime-wasm-optional runtime-nativeaot-outerloop runtime-coreclr jitstress-isas-sve

jkotas · 2024-08-13T19:40:53Z

/azp run runtime-libraries-coreclr outerloop

azure-pipelines · 2024-08-13T19:41:07Z

Azure Pipelines successfully started running 1 pipeline(s).

jkotas · 2024-08-13T19:46:31Z

Is this change even getting triggered by this PR run? I'm guessing not?

Correct. The new config is in the extra platforms section that is not triggered by default on PRs.

richlander · 2024-08-14T01:37:08Z

It isn't obvious to me that this worked. I searched for "azurelinux" on this page (where other helix images are listed) and see no hits.

https://dev.azure.com/dnceng-public/public/_build/results?buildId=774725&view=logs&j=df56ac33-e9fe-5eb2-5cdb-3303d7aad000&t=f62247f8-d051-5cb6-d5f0-06f532f9a005

@sbomer was suggesting that runtime-extra-platforms might be the correct choice. Was that right Sven?

jkotas · 2024-08-14T02:21:45Z

/azp run runtime-extra-platforms

azure-pipelines · 2024-08-14T02:22:05Z

Azure Pipelines successfully started running 1 pipeline(s).

richlander · 2024-08-14T04:24:34Z

That appears to have worked. Thanks! Now have to sort through the failures.

From linux-x64 Release Libraries_Release_CoreCLR. See last line.

  Job 9dd74d98-088e-420c-ac29-bd09c3137c32 on SLES.15.Amd64.Open is completed with 252 finished work items.
  Job de58e71a-b7f7-47e4-a224-6256e6481fcb on Ubuntu.2204.Amd64.Open is completed with 252 finished work items.
  Job 4105daeb-2110-4c8a-a440-abaa51adbd4c on (Debian.11.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:debian-11-helix-amd64 is completed with 252 finished work items.
  Job 3172cef6-ff63-4ccf-974b-4ed486ac49fc on (Fedora.38.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:fedora-38-helix is completed with 252 finished work items.
  Job 72282a81-4343-418b-b29c-6f912b0ce732 on (Mariner.2.0.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:cbl-mariner-2.0-helix-amd64 is completed with 252 finished work items.
  Job 930393ef-df5e-4161-813e-d12cb38af42b on (AzureLinux.3.0.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:azurelinux-3.0-helix-amd64 is completed with 252 finished work items.

richlander · 2024-08-14T20:23:41Z

/root/helix/work/workitem/e /root/helix/work/workitem/e
  Discovering: System.Net.Quic.Functional.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.Quic.Functional.Tests (found 4 of 152 test cases)
  Starting:    System.Net.Quic.Functional.Tests (parallel test collections = on [2 threads], stop on fail = off)
MsQuic not supported and using 'unknown'.
    System.Net.Quic.Tests.MsQuicPlatformDetectionTests.SupportedLinuxPlatformsWithMsQuic_IsSupportedIsTrue [FAIL]
      Assert.True() Failure
      Expected: True
      Actual:   False
      Stack Trace:
        /_/src/libraries/System.Net.Quic/tests/FunctionalTests/MsQuicPlatformDetectionTests.cs(51,0): at System.Net.Quic.Tests.MsQuicPlatformDetectionTests.SupportedLinuxPlatformsWithMsQuic_IsSupportedIsTrue()
        --- End of stack trace from previous location ---
      Output:
        /usr/lib/libmsquic.so.2
        /usr/lib/libmsquic.so
        /usr/lib/libmsquic.so.2.4.3

We added libmsquic to the image. Does it need to learn about Azure Linux? I'm looking at using 'unknown'..

@liveans @wfurt

liveans · 2024-08-14T20:31:13Z

We added libmsquic to the image. Does it need to learn about Azure Linux? I'm looking at using 'unknown'..

@liveans @wfurt

I'll take a look what's wrong, might be missing dependency.

wfurt · 2024-08-14T20:37:18Z

helixbot [ / ]$ ldd /usr/lib/libmsquic.so.2.4.3
	linux-vdso.so.1 (0x00007ffc131ef000)
	libcrypto.so.3 => /usr/lib/libcrypto.so.3 (0x00007f3f60061000)
	libatomic.so.1 => not found
	libnuma.so.1 => not found
	libc.so.6 => /usr/lib/libc.so.6 (0x00007f3f5fe1e000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f3f606ba000)

liveans · 2024-08-14T20:37:37Z

It is indeed missing dependency.

After installing libnuma1, libatomic1 should work.

/cc @richlander

wfurt · 2024-08-14T20:45:08Z

we should update the pattern in https://github.com/dotnet/dotnet-buildtools-prereqs-docker/ to verify that produced library at least loads. With that, one could find issues before going all the way to runtime.

liveans · 2024-08-14T20:48:35Z

we should update the pattern in https://github.com/dotnet/dotnet-buildtools-prereqs-docker/ to verify that produced library at least loads. With that, one could find issues before going all the way to runtime.

We can actually run msquictest --gtest_filter=ParameterValidation.ValidateApi, that's something msquic itself using on Packaging pipeline to verify package is loading or not.

jkotas · 2024-08-14T20:55:59Z

We can do all that, but I think we are missing the user point of view.

If our users want to use msquic on Azure Linux, how are they expected to get it? Are they expected to build their own like what we are doing in dotnet-buildtools-prereqs-docker? Should it be documented somewhere?

richlander · 2024-08-14T20:57:24Z

FYI: microsoft/msquic#4450

liveans · 2024-08-14T20:57:59Z

We can do all that, but I think we are missing the user point of view.

If our users want to use msquic on Azure Linux, how are they expected to get it? Are they expected to build their own like what we are doing in dotnet-buildtools-prereqs-docker? Should it be documented somewhere?

msquic team is working on it: microsoft/msquic#4455, they are going to publish packages soon.

jkotas · 2024-08-14T21:05:17Z

msquic team is working on it: microsoft/msquic#4455, they are going to publish packages soon.

Should we just wait for the proper msquic package to get published?

If we cannot wait, we should make sure to revert dotnet/dotnet-buildtools-prereqs-docker#1177 workaround once the proper package is available.

azure-pipelines · 2024-08-28T12:27:19Z

Azure Pipelines successfully started running 1 pipeline(s).

vcsjones · 2024-08-28T20:15:54Z

Looks like test artifacts are failing to publish, but looking at the raw logs from System.Security.Cryptography, the output is promising.

https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-106330-merge-12fcefff9a154ce5ae/System.Security.Cryptography.Tests/1/console.a7858ece.log?helixlogtype=result

=== TEST EXECUTION SUMMARY ===
System.Security.Cryptography.Tests Total: 9776, Errors: 0, Failed: 0, Skipped: 310, Time: 106.457s

vcsjones · 2024-08-28T20:25:00Z

Hm, that may be the wrong breadcrumb of test results. I don't see them for AZL.

richlander · 2024-08-28T21:01:04Z

Thoughts @carlossanlop?

carlossanlop · 2024-08-28T21:22:48Z

There was a helix outage reported for most of the day today. I think a retry from scratch should work. Let's close and reopen this, and execute runtime-extra-platforms once again.

carlossanlop · 2024-08-28T21:23:24Z

/azp run runtime-extra-platforms

azure-pipelines · 2024-08-28T21:23:36Z

Azure Pipelines successfully started running 1 pipeline(s).

richlander · 2024-08-28T23:36:28Z

Looks like Azure Linux is coming out clean (except the System.IO.Tests). Is that your read @vcsjones?

bartonjs · 2024-08-28T23:37:01Z

https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-106330-merge-e655bc6eb742442c9c/System.Security.Cryptography.Pkcs.Tests/1/console.c243fbb9.log?helixlogtype=result

Looks like there's still a test in the S.S.C.Pkcs library using RSA+MD5:

runtime/src/libraries/System.Security.Cryptography.Pkcs/tests/Pkcs12/KeyBagTests.cs

Lines 29 to 42 in 063ae05

    
           Assert.True(rsa2.TrySignData( 
        
               keyBag.Pkcs8PrivateKey.Span, 
        
               sig, 
        
               HashAlgorithmName.MD5, 
        
               RSASignaturePadding.Pkcs1, 
        
               out int sigLen)); 
        
           Assert.Equal(sig.Length, sigLen); 
        
           Assert.True(rsa.VerifyData( 
        
               keyBag.Pkcs8PrivateKey.Span, 
        
               sig, 
        
               HashAlgorithmName.MD5, 
        
               RSASignaturePadding.Pkcs1));

vcsjones · 2024-08-29T02:09:49Z

/azp run runtime-extra-platforms

azure-pipelines · 2024-08-29T02:10:08Z

Azure Pipelines successfully started running 1 pipeline(s).

richlander · 2024-08-29T05:37:31Z

That looks clean now. Yes?

vcsjones · 2024-08-29T13:55:39Z

That looks clean now. Yes?

Yes. All the linux legs look green to me. The only failures I see are Apple iOS / tvOS / Catalyst which are unrelated.

richlander · 2024-08-29T14:01:11Z

Sounds good. We just need an approval now. We can then decide how to approach release/8.0-staging. The current plan is to not offer Azure Linux 3.0 support for .NET 6.

vcsjones · 2024-08-29T14:03:37Z

We can then decide how to approach release/8.0-staging.

We also need to backport to release/9.0.

release/9.0 has one more pending backport for a green crypto test run. [release/9.0] Azure Linux 3.0 and test fixes #107123
release/8.0-staging has a backport PR of all fixes rolled up ready to go. [release/8.0-staging] Backport Azure Linux test changes #107061

richlander · 2024-08-29T14:13:41Z

Would you consider adding the pipeline change to those PRs so that you can run the runtime-extra-platforms pipeline just before merging (and then we're done?).

https://github.com/dotnet/runtime/pull/106330/files#diff-cbc8f55c9e8b4fc53ff194be8f6e99b461bbb888220a8ede2e5edb53b947b287R71

vcsjones · 2024-08-29T14:31:03Z

Would you consider adding the pipeline change to those PRs so that you can run the runtime-extra-platforms pipeline just before merging (and then we're done?).

https://github.com/dotnet/runtime/pull/106330/files#diff-cbc8f55c9e8b4fc53ff194be8f6e99b461bbb888220a8ede2e5edb53b947b287R71

I cherry picked this PR onto those backport PRs and kicked off a run.

richlander · 2024-08-29T14:40:32Z

Can you offer us an approval @bartonjs?

richlander · 2024-08-29T17:06:58Z

Thanks @bartonjs and @vcsjones. Much appreciated. That was all quite unexpected.

Co-authored-by: Kevin Jones <[email protected]>

Add Azure 3.0 helix test

1528973

dotnet-issue-labeler bot added the area-Infrastructure-libraries label Aug 13, 2024

Merge branch 'main' into azure-linux-helix

a7c0b16

dotnet-policy-service bot assigned richlander Aug 13, 2024

richlander mentioned this pull request Aug 13, 2024

Tracking: Add Azure Linux for build and test #106331

Open

This was referenced Aug 13, 2024

The Operation will be canceled. The next steps may not contain expected logs. dotnet/dnceng#3008

Open

WasmTestOnChrome timeouts in CI #105363

Closed

richlander mentioned this pull request Aug 14, 2024

Add libnuma and libatomic dotnet/dotnet-buildtools-prereqs-docker#1177

Merged

carlossanlop closed this Aug 28, 2024

carlossanlop reopened this Aug 28, 2024

build-analysis bot mentioned this pull request Aug 29, 2024

SIGKILL (OOM?) while running LibraryImportGenerator.Tests w/o actionable log messages or artifacts dotnet/dnceng#2496

Open

3 tasks

Merge branch 'main' into azure-linux-helix

bc7652a

This was referenced Aug 29, 2024

[release/9.0] Azure Linux 3.0 and test fixes #107123

Merged

[release/8.0-staging] Backport Azure Linux test changes #107061

Merged

bartonjs approved these changes Aug 29, 2024

View reviewed changes

richlander merged commit 1b656fe into dotnet:main Aug 29, 2024
112 of 122 checks passed

richlander deleted the azure-linux-helix branch August 29, 2024 17:07

jtschuster pushed a commit to jtschuster/runtime that referenced this pull request Sep 17, 2024

Add Azure 3.0 helix test (dotnet#106330)

435067a

Co-authored-by: Kevin Jones <[email protected]>

github-actions bot locked and limited conversation to collaborators Oct 2, 2024

Add Azure 3.0 helix test #106330

Add Azure 3.0 helix test #106330

Conversation

richlander commented Aug 13, 2024

richlander commented Aug 13, 2024 • edited Loading

jkotas commented Aug 13, 2024

azure-pipelines bot commented Aug 13, 2024

jkotas commented Aug 13, 2024

azure-pipelines bot commented Aug 13, 2024

jkotas commented Aug 13, 2024

azure-pipelines bot commented Aug 13, 2024

jkotas commented Aug 13, 2024

richlander commented Aug 14, 2024

jkotas commented Aug 14, 2024

azure-pipelines bot commented Aug 14, 2024

richlander commented Aug 14, 2024 • edited Loading

richlander commented Aug 14, 2024

liveans commented Aug 14, 2024

wfurt commented Aug 14, 2024

liveans commented Aug 14, 2024

wfurt commented Aug 14, 2024

liveans commented Aug 14, 2024

jkotas commented Aug 14, 2024

richlander commented Aug 14, 2024

liveans commented Aug 14, 2024 • edited Loading

jkotas commented Aug 14, 2024

azure-pipelines bot commented Aug 28, 2024

vcsjones commented Aug 28, 2024

vcsjones commented Aug 28, 2024

richlander commented Aug 28, 2024

carlossanlop commented Aug 28, 2024

carlossanlop commented Aug 28, 2024

azure-pipelines bot commented Aug 28, 2024

richlander commented Aug 28, 2024

bartonjs commented Aug 28, 2024

vcsjones commented Aug 29, 2024

azure-pipelines bot commented Aug 29, 2024

richlander commented Aug 29, 2024

vcsjones commented Aug 29, 2024

richlander commented Aug 29, 2024

vcsjones commented Aug 29, 2024 • edited Loading

richlander commented Aug 29, 2024

vcsjones commented Aug 29, 2024

richlander commented Aug 29, 2024

richlander commented Aug 29, 2024

richlander commented Aug 13, 2024 •

edited

Loading

richlander commented Aug 14, 2024 •

edited

Loading

liveans commented Aug 14, 2024 •

edited

Loading

vcsjones commented Aug 29, 2024 •

edited

Loading