Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Wasm] Add AOT CI Leg #46651

Merged
merged 19 commits into from
Jan 20, 2021
Merged

[Wasm] Add AOT CI Leg #46651

merged 19 commits into from
Jan 20, 2021

Conversation

steveisok
Copy link
Member

@steveisok steveisok commented Jan 6, 2021

Contributes to #46633

With the EnableAggressiveTrimming flag added in #46367, the library testing size has been reduced to a much smaller size (~2MB), making AOT compilation of these library tests more reasonable. This PR stands-up a lane on CI to see how far wasm AOT library tests can go.

A significant number of library tests have failed on CI. Most of them have failed because they do not include Microsoft.DotNet.RemoteExecutor, which had been added to https://github.com/dotnet/runtime/blob/master/eng/testing/ILLink.Descriptor.xunit.xml due to a linker bug dotnet/linker#1713.

The failures in the lane have been parsed out to the following test suites that are missing Microsoft.DotNet.RemoteExecutor.

The test suites above have been re-ran locally with the exclusion of Microsoft.DotNet.RemoteExecutor to verify whether the test fails or passes, and suites continuing to fail have been added to project exclusions in tests.proj. A significant number of test suites fall under several categories of failures, but for simplicity, they are listed under just one, and issues have been created for each unique failure.

In lieu of running all the library tests suites in AOT mode on CI (estimated to be ~36 hrs = 240 test suites * ~9min AOT time per test suite), only System.Buffers.Tests.csproj will be ran in AOT mode to standup the leg. In the meantime, alternative solutions are being worked on.

The CI lane will be located in runtime-staging.yml to avoid blocking and failing builds.

@steveisok steveisok added arch-wasm WebAssembly architecture area-Infrastructure-mono labels Jan 6, 2021
@ghost
Copy link

ghost commented Jan 6, 2021

Tagging subscribers to this area: @directhex
See info in area-owners.md if you want to be subscribed.

Issue Details
Author: steveisok
Assignees: -
Labels:

arch-wasm, area-Infrastructure-mono

Milestone: -

#
# Build the whole product using Mono and run libraries tests, multi-scenario
#
- template: /eng/pipelines/common/platform-matrix.yml
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would suggest adding this to runtime-staging once this is ready to be merged (I know we're on early process gathering info on how many tests fail, etc) to get it stable.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The lane passed after running just System.Buffers.Tests.csproj :)
https://dev.azure.com/dnceng/public/_build/results?buildId=948229&view=logs&j=f217f715-90ad-5404-53bd-b93e3b80f465&t=9df76bd4-3041-5537-002b-9e29a53263b9
Moved the block to runtime-staging.yml

@steveisok steveisok merged commit ecc0d8a into dotnet:master Jan 20, 2021
@steveisok steveisok deleted the wasm-aot-ci-leg branch January 20, 2021 17:02
@ghost ghost locked as resolved and limited conversation to collaborators Feb 19, 2021

<!-- Run only a small randomly chosen set of passing test suites on CI because running all suites is expected to take longer than a day -->
<ProjectExclusions Include="$(MSBuildThisFileDirectory)*\tests\**\*.Tests.csproj" />
<ProjectExclusions Remove="$(MSBuildThisFileDirectory)System.Buffer.Tests\tests\System.Buffer.Tests.csproj" />
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@steveisok Btw, the $(MSBuildThisFileDirectory)System.Buffer.Tests directory doesn't exist, so this is not testing anything.

Suggested change
<ProjectExclusions Remove="$(MSBuildThisFileDirectory)System.Buffer.Tests\tests\System.Buffer.Tests.csproj" />
<ProjectExclusions Remove="$(MSBuildThisFileDirectory)System.Buffer\tests\System.Buffer.Tests.csproj" />

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the catch! There is more work being done in #48226 that removes this line, and later the exclusions will be revisited and updated according to this tracking issue #48079.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants