[Wasm] Tracking AOT wasm library test failures

[mdh1418]

To see where the wasm library tests stand with AOT compilation with respect to how many pass/fail, what the failures are, how many hang, the 268 test suites were ran locally.

Dropbox paper with findings

All CI Results

Among all the test suites failing on CI, the following are the main unique test failures observed.

---

Unique Test Failures based on CI logs

• a) Assert.True() Failure [Wasm][AOT] Assert.True() failure #49246
• b) System.ExecutionEngineException : AOT Compilation failed for method wasm AOT library test failure - AOT Compilation failed for method #49219
• c) System.ExecutionEngineException : Attempting to JIT compile method wasm AOT library test failure - Attempting to JIT compile method #46708
• d) console.error: RuntimeError: memory access out of bounds wasm AOT library test failure - memory access out of bounds #46700
• e) System.NotImplementedException : The method or operation is not implemented. [Wasm][AOT] System.NotImplementedException : The method or operation is not implemented. #49229
• f) System.ExecutionEngineException : AOT Compilation failed for method 'void System.Threading.CancellationTokenSource:ExecuteCallbackHandlers (bool)'. wasm AOT library test failure - AOT Compilation failed for method #49219
• g) System.ExecutionEngineException : AOT Compilation failed for method 'bool Microsoft.Extensions.Internal.ParameterDefaultValue:TryGetDefaultValue (System.Reflection.ParameterInfo,object&)'. wasm AOT library test failure - AOT Compilation failed for method #49219
• h) System.ExecutionEngineException : Attempting to JIT compile method '(wrapper dynamic-method) object Microsoft.Extensions.DependencyInjection.ServiceLookup.ILEmitResolverBuilder:ResolveService (Microsoft.Extensions.DependencyInjection.ServiceLookup.ILEmitResolverBuilder/ILEmitResolverBuilderRuntimeContext,Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope)' while running in aot-only mode. See https://docs.microsoft.com/xamarin/ios/internals/limitations for more information. wasm AOT library test failure - Attempting to JIT compile method #46708
• i) System.ExecutionEngineException : Attempting to JIT compile method '(wrapper dynamic-method) object object:lambda_method wasm AOT library test failure - Attempting to JIT compile method #46708
• j) AOT Compilation failed for method 'object Microsoft.Extensions.DependencyInjection.ActivatorUtilities/ConstructorMatcher:CreateInstance (System.IServiceProvider)'. wasm AOT library test failure - AOT Compilation failed for method #49219
• k) System.ExecutionEngineException : AOT Compilation failed for method 'System.Threading.Tasks.Task System.Net.Http.HttpContent:LoadIntoBufferAsync (long,System.Threading.CancellationToken)'. wasm AOT library test failure - AOT Compilation failed for method #49219
• l) EventName and ExceptionJson is supposed to be null but it isn't. [Wasm][AOT] Microsoft.Extensions.Logging.EventSource.Tests failures #49240
• m) LoggerName is supposed to be null but it isn't. [Wasm][AOT] Microsoft.Extensions.Logging.EventSource.Tests failures #49240
• n) console.error: RuntimeError: invalid index into function table wasm AOT library test failure - invalid index into function table #46731
• p) System.Composition.Hosting.CompositionFailedException : No export was found for the contract [Wasm][AOT] System.Composition.Hosting.CompositionFailedException #49230
• q) System.Composition.Hosting.CompositionFailedException : No importing constructor was found on type [Wasm][AOT] System.Composition.Hosting.CompositionFailedException #49230
• u) System.ExecutionEngineException : AOT Compilation failed for method 'void System.Data.DataColumnCollection:BaseAdd (System.Data.DataColumn)'. wasm AOT library test failure - AOT Compilation failed for method #49219
• v) System.ExecutionEngineException : AOT Compilation failed for method 'void System.Data.DataRow:set_Item (System.Data.DataColumn,object)' wasm AOT library test failure - AOT Compilation failed for method #49219
• w) System.IO.DirectoryNotFoundException : Could not find a part of the path or System.IO.FileNotFoundException : Could not find file [Wasm][AOT] System.IO.DirectoryNotFoundException And System.IO.FileNotFoundException #49232 (@fanyang-mono working on this)
• x) Assert.NotEmpty() Failure [Wasm][AOT] System.Diagnostics.TraceSource.Tests failures #49241
• y) Assert.Contains() Failure [Wasm][AOT] System.Diagnostics.TraceSource.Tests failures #49241
• z) System.ExecutionEngineException : AOT Compilation failed for method 'void System.IO.MemoryMappedFiles.Tests.MemoryMappedFileTests_CreateFromFile wasm AOT library test failure - AOT Compilation failed for method #49219
• ab) System.ExecutionEngineException : AOT Compilation failed for method 'void System.Net.Http.HttpContent:CopyTo (System.IO.Stream,System.Net.TransportContext,System.Threading.CancellationToken)'. wasm AOT library test failure - AOT Compilation failed for method #49219
  Special case (Invariant.Tests)
• ac) System.ExecutionEngineException : AOT Compilation failed for method 'void System.Net.Mime.ContentType:ParseValue ()'. wasm AOT library test failure - AOT Compilation failed for method #49219
• ad) [byte difference] [Wasm][AOT] System.Reflection.Metadata.Tests failures #49245
• ae) System.InvalidOperationException : Failed to compare two elements in the array. wasm AOT library test failure - System.InvalidOperationException #46759
• ag) Assert.Throws() Failure [Wasm][AOT] System.Runtime.CompilerServices.Unsafe.Tests failures #49242
• aj) console.error: RuntimeError: function signature mismatch wasm AOT library test failure - function signature mismatch #46761 log
• ak) System.NullReferenceException : Object reference not set to an instance of an object. [Wasm][AOT] System.NullReferenceException : Object reference not set to an instance of an object. #49234
• al) System.ExecutionEngineException : AOT Compilation failed for method 'void System.Numerics.Tests.divremTest:VerifyDivRemString (string)'. wasm AOT library test failure - AOT Compilation failed for method #49219
• am) Assertion at /__w/1/s/src/mono/mono/metadata/class-init.c:2746, condition '<disabled>' not met wasm AOT library test failure - condition <disabled> not met #46698 log
• an) Assertion: should not be reached at /__w/1/s/src/mono/mono/metadata/class-accessors.c:86 [Wasm][AOT] Assertion: should not be reached at src/mono/mono/metadata/class-accessors.c:83 #45304
• ao) Assertion at /__w/1/s/src/mono/mono/mini/mini-generic-sharing.c:2168, condition '<disabled>' not met wasm AOT library test failure - condition <disabled> not met #46698 log
• ap) Process 24507 didn't exit within 00:15:00 and will be killed [Wasm][AOT] System.Threading.Tasks.Dataflow.Tests failures #49243
• aq) Assertion at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:3965, condition '<disabled>' not met wasm AOT library test failure - condition <disabled> not met #46698 log
• as) console.error: Unhandled exception in _finalize_startup Error: FS error [Wasm][AOT] System.IO.FileSystem.Tests failures #49244
• au) Assertion: should not be reached at /__w/1/s/src/mono/mono/mini/../sgen/sgen-scan-object.h:91 [Wasm][AOT] Assertion: should not be reached at /__w/1/s/src/mono/mono/mini/../sgen/sgen-scan-object.h:91 #49225

EnableAggressiveTrimming related failures
r) System.ArgumentException : Type Moq.Internals.InterfaceProxy is not valid base type for interface proxy, because it does not have accessible parameterless constructor. Only a non-sealed class with non-private default constructor can be used as base type for interface proxy. Please use some other valid type. #46760 log
s) System.InvalidOperationException : Sequence contains no matching element #46759
t) System.ArgumentNullException : Value cannot be null. (Parameter 'method') #46758 (@naricc working on this)
af) System.NotSupportedException : Use of ResourceManager for custom types is disabled. Set the MSBuild Property CustomResourceTypesSupport to true in order to enable it.
ah) console.error: System.AggregateException: One or more errors occurred. (Object reference not set to an instance of an object.) ---> System.NullReferenceException: Object reference not set to an instance of an object.
ai) Could not load file or assembly 'System.Runtime, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies log

---

The Unique failures may appear across several test suites. Each test suite below has failed on CI, and are tagged with the corresponding failure by the alphabetical tag.

Suites that did not produce test results

---

Error: Failed to load method 0x6000001

• System.ComponentModel.Composition.Tests -- ai
• System.Diagnostics.Tracing.Tests -- ai
• System.Runtime.Loader.DefaultContext.Tests -- ai
• System.Runtime.Loader.RefEmitLoadContext.Tests -- ai
• System.Runtime.Tests -- Precompiling failed for log

---

Suites that failed with 1 or more test failures

• System.Data.Common.Tests -- u
• System.Security.Cryptography.Algorithms.Tests -- aj
• System.Numerics.Vectors.Tests -- c
• System.Linq.Queryable.Tests -- i, d
• System.IO.FileSystem.Tests -- as
• System.Linq.Expressions.Tests -- an
• System.Runtime.Serialization.Xml.Tests -- an
• Microsoft.Extensions.DependencyInjection.Tests -- g, h, i, j, k
• System.Runtime.Serialization.Json.Tests -- c
• System.Composition.Tests -- i, p, q
• System.Xml.XmlSchema.XmlSchemaValidatorApi.Tests -- w
• System.ComponentModel.TypeConverter.Tests -- i, d, t
• System.Threading.Channels.Tests -- c, f
• System.IO.Pipelines.Tests -- f, y
• System.Data.DataSetExtensions.Tests -- u, v, w, missing 33 tests
• System.Runtime.Numerics.Tests -- ak, al
• System.ServiceModel.Syndication.Tests -- c, w
• System.Collections.Immutable.Tests -- d
• System.ComponentModel.Composition.Registration.Tests -- i
• System.Reflection.DispatchProxy.Tests -- c
• System.Net.Http.Json.Functional.Tests -- c, f
• Microsoft.Extensions.Configuration.Json.Tests -- a, f
• Microsoft.Extensions.Logging.EventSource.Tests -- g, l, m
• Microsoft.Extensions.Configuration.Functional.Tests -- f
• System.IO.Compression.Tests -- w
• Common.Tests -- a, b
• System.Collections.Concurrent.Tests -- e, f
• System.IO.MemoryMappedFiles.Tests -- z
• System.Composition.Convention.Tests -- i
• System.Threading.Tests -- f
• System.Xml.XmlSchemaSet.Tests -- a, w
• Microsoft.Extensions.Primitives.Tests -- f, r
• System.Composition.TypedParts.Tests -- i
• Microsoft.Extensions.Http.Tests -- i, k, t
• System.Reflection.Emit.Lightweight.Tests -- c
• Microsoft.Extensions.Configuration.Ini.Tests -- a, f
• System.IO.Tests -- Random text Kæ￿ $@#�'ã†�ìŽ A5 not AOT related
• Microsoft.Extensions.Configuration.Xml.Tests -- a, f
• System.Collections.Tests -- e, killed
• System.Text.RegularExpressions.Tests -- au
• System.CodeDom.Tests -- d
• System.Drawing.Primitives.Tests -- c
• System.Memory.Tests -- c
• System.ObjectModel.Tests -- d
• Microsoft.Extensions.Options.Tests -- f
• Microsoft.Extensions.Configuration.Tests -- f
• Microsoft.Extensions.Configuration.UserSecrets.Tests -- f
• Microsoft.Extensions.Logging.Testing.Tests -- g
• System.IO.Packaging.Tests -- w
• System.Memory.Data.Tests -- f, c
• System.Net.Primitives.Functional.Tests -- e
• Microsoft.Extensions.FileProviders.Composite.Tests -- a, r
• System.Diagnostics.TraceSource.Tests -- x, y
• Microsoft.Extensions.Configuration.Binder.Tests -- e
• Microsoft.Extensions.Configuration.FileExtensions.Tests -- r
• System.Composition.Hosting.Tests -- i
• System.Globalization.Tests -- e
• System.Net.WebSockets.Tests -- c
• System.Reflection.CoreCLR.Tests -- EnableAggressiveTrimming failure
• System.Transactions.Local.Tests -- ak
• System.Diagnostics.DiagnosticSource.Tests -- t
• System.Net.Http.Json.Unit.Tests -- c
• System.Net.Http.Unit.Tests -- k, ab
• System.Resources.Reader.Tests -- af
• System.Runtime.CompilerServices.Unsafe.Tests -- ag
• System.Threading.Tasks.Extensions.Tests -- s
• System.Reflection.MetadataLoadContext.Tests -- 360 failures -- missing 78 tests -- w
• System.Diagnostics.StackTrace.Tests -- 22 failures -- a
• System.Reflection.Tests -- missing ~1100 tests -- d
• Microsoft.CSharp.Tests -- c, d
• Microsoft.Extensions.Logging.Tests -- n
• Microsoft.VisualBasic.Core.Tests -- am
• System.Dynamic.Runtime.Tests -- an
• System.Linq.Tests -- i
• System.Net.Mail.Functional.Tests -- ac
• System.Private.Runtime.InteropServices.JavaScript.Tests -- ao
• System.Reflection.Context.Tests -- am
• System.Reflection.Emit.ILGeneration.Tests -- c
• System.Reflection.Emit.Tests -- c
• System.Reflection.Metadata.Tests -- c, w, ad, ae
• System.Runtime.Extensions.Tests -- d
• System.Runtime.InteropServices.Tests -- c, ah
• System.Runtime.Loader.Tests -- am
• System.Runtime.Serialization.Json.ReflectionOnly.Tests -- ao
• System.Runtime.Serialization.Xml.ReflectionOnly.Tests -- ao
• System.Text.Json.Tests -- b, c, d
• System.Threading.Tasks.Dataflow.Tests -- ap
• System.Threading.Tasks.Parallel.Tests -- f
• System.Threading.Tasks.Tests -- d
• System.ValueTuple.Tests -- aj
• System.Xml.Linq.xNodeBuilder.Tests -- aq
• System.Xml.XmlSerializer.ReflectionOnly.Tests -- i
• System.Xml.XmlSerializer.Tests -- ao
• System.Xml.Xsl.XslCompiledTransformApi.Tests -- w
• System.Xml.Xsl.XslTransformApi.Tests -- w

Tests failing in CI

• System.Collections.Specialized.Tests -- d
• System.IO.Compression.ZipFile.Tests -- w
• System.Private.Uri.Functional.Tests -- n
• System.Xml.RW.XmlWriterApi.Tests -- aj
• System.Xml.XPath.XmlDocument.Tests -- aj
• System.Xml.XmlDocument.Tests -- w
• System.Xml.XmlResolver.Tests -- w
• System.Xml.RW.XmlSystemPathResolver.Tests -- w

[ghost]

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

---

To see where the wasm library tests stand with AOT compilation with respect to how many pass/fail, what the failures are, how many hang, the 268 test suites were ran locally.

Dropbox paper with findings

---

Suites that did not produce wasm-console.log or testResults.xml (48 suites)

• Microsoft.Diagnostics.Tracing.EventSource.Redist.Tests
• Microsoft.Win32.Registry.AccessControl.Tests
• Microsoft.Win32.Registry.Tests
• Microsoft.Win32.SystemEvents.Tests
• System.ComponentModel.Composition.Tests
• System.Configuration.ConfigurationManager.Tests
• System.Data.Odbc.Tests
• System.Data.OleDb.Tests
• System.Diagnostics.Debug.Tests
• System.Diagnostics.EventLog.Tests
• System.Diagnostics.FileVersionInfo.Tests
• System.Diagnostics.PerformanceCounter.Tests
• System.Diagnostics.StackTrace.Tests
• System.Diagnostics.Tracing.Tests
• System.DirectoryServices.AccountManagement.Tests
• System.DirectoryServices.Protocols.Tests
• System.DirectoryServices.Tests
• System.Drawing.Common.Tests
• System.Globalization.Extensions.Nls.Tests
• System.Globalization.Nls.Tests
• System.IO.FileSystem.AccessControl.Tests
• System.IO.FileSystem.Watcher.Tests
• System.IO.Pipes.AccessControl.Tests
• System.IO.Ports.Tests
• System.Management.Tests
• System.Net.Http.WinHttpHandler.Functional.Tests
• System.Net.Http.WinHttpHandler.Unit.Tests
• System.Net.NameResolution.Pal.Tests
• System.Net.Quic.Functional.Tests
• System.Reflection.MetadataLoadContext.Tests
• System.Reflection.Tests
• System.Resources.Extensions.Tests
• System.Resources.ResourceManager.Tests
• System.Runtime.Caching.Tests
• System.Runtime.Loader.DefaultContext.Tests
• System.Runtime.Loader.RefEmitLoadContext.Tests
• System.Runtime.Nls.Tests
• System.Runtime.Serialization.Formatters.Tests
• System.Runtime.Tests
• System.Security.AccessControl.Tests
• System.Security.Cryptography.Cng.Tests
• System.Security.Cryptography.ProtectedData.Tests
• System.Security.Permissions.Tests
• System.Security.Principal.Windows.Tests
• System.ServiceProcess.ServiceController.Tests
• System.Speech.Tests
• System.Threading.AccessControl.Tests
• System.Windows.Extensions.Tests

---

Suites that did not run to completion (did not produce testResults.xml (33 suites)

• Microsoft.CSharp.Tests
• Microsoft.Extensions.Caching.Memory.Tests
• Microsoft.Extensions.DependencyModel.Tests
• Microsoft.Extensions.Hosting.Unit.Tests
• Microsoft.Extensions.Logging.Tests
• Microsoft.VisualBasic.Core.Tests
• System.Console.Tests
• System.Dynamic.Runtime.Tests
• System.IO.IsolatedStorage.Tests
• System.Linq.Parallel.Tests
• System.Linq.Tests
• System.Net.Http.Functional.Tests
• System.Net.Mail.Functional.Tests
• System.Private.Runtime.InteropServices.JavaScript.Tests
• System.Reflection.Context.Tests
• System.Reflection.Emit.ILGeneration.Tests
• System.Reflection.Emit.Tests
• System.Reflection.Metadata.Tests
• System.Runtime.Extensions.Tests
• System.Runtime.InteropServices.Tests
• System.Runtime.Loader.Tests
• System.Runtime.Serialization.Json.ReflectionOnly.Tests
• System.Runtime.Serialization.Xml.ReflectionOnly.Tests
• System.Text.Json.Tests
• System.Threading.Tasks.Dataflow.Tests
• System.Threading.Tasks.Parallel.Tests
• System.Threading.Tasks.Tests
• System.ValueTuple.Tests
• System.Xml.Linq.xNodeBuilder.Tests
• System.Xml.XmlSerializer.ReflectionOnly.Tests
• System.Xml.XmlSerializer.Tests
• System.Xml.Xsl.XslCompiledTransformApi.Tests
• System.Xml.Xsl.XslTransformApi.Tests

---

Suites that failed with 1 or more test failures (descending in number of failures) (103 suites)

• System.Net.Sockets.Tests
• System.Data.Common.Tests
• System.Security.Cryptography.Algorithms.Tests
• System.Numerics.Vectors.Tests
• System.Security.Cryptography.OpenSsl.Tests
• System.Security.Cryptography.Pkcs.Tests
• System.Security.Cryptography.X509Certificates.Tests
• System.Security.Cryptography.Csp.Tests
• System.Linq.Queryable.Tests
• Microsoft.Extensions.DependencyInjection.ExternalContainers.Tests
• System.IO.FileSystem.Tests
• System.Linq.Expressions.Tests
• System.Runtime.Serialization.Xml.Tests
• Microsoft.Extensions.DependencyInjection.Tests
• System.Net.HttpListener.Tests
• System.Security.Cryptography.Xml.Tests
• System.Runtime.Serialization.Json.Tests
• System.Diagnostics.Process.Tests
• System.Security.Cryptography.Encoding.Tests
• System.Composition.Tests
• System.Net.Requests.Tests
• System.Xml.XmlSchema.XmlSchemaValidatorApi.Tests
• System.ComponentModel.TypeConverter.Tests
• System.Net.NetworkInformation.Functional.Tests
• Microsoft.Extensions.FileProviders.Physical.Tests
• System.Threading.Overlapped.Tests
• System.Threading.Channels.Tests
• System.IO.Pipelines.Tests
• System.Net.NameResolution.Functional.Tests
• System.Data.DataSetExtensions.Tests
• System.Runtime.Numerics.Tests
• System.ServiceModel.Syndication.Tests
• System.IO.Compression.Brotli.Tests
• System.Net.WebClient.Tests
• System.Collections.Immutable.Tests
• System.ComponentModel.Composition.Registration.Tests
• System.Reflection.DispatchProxy.Tests
• System.Net.Http.Json.Functional.Tests
• System.Net.Ping.Functional.Tests
• Microsoft.Extensions.Configuration.Json.Tests
• Microsoft.Extensions.Logging.EventSource.Tests
• System.Runtime.InteropServices.RuntimeInformation.Tests
• Microsoft.Extensions.Configuration.Functional.Tests
• System.IO.Compression.Tests
• Common.Tests
• System.Collections.Concurrent.Tests
• System.IO.MemoryMappedFiles.Tests
• System.Composition.Convention.Tests
• System.Text.Encoding.Tests
• System.Threading.Tests
• System.Xml.XmlSchemaSet.Tests
• Microsoft.Extensions.Primitives.Tests
• System.Composition.TypedParts.Tests
• Microsoft.Extensions.Http.Tests
• System.Reflection.Emit.Lightweight.Tests
• Microsoft.Extensions.Configuration.Ini.Tests
• System.IO.Tests
• Microsoft.Extensions.Configuration.Xml.Tests
• System.Security.Cryptography.Primitives.Tests
• System.Collections.NonGeneric.Tests
• System.Collections.Tests
• System.Text.RegularExpressions.Tests
• System.CodeDom.Tests
• System.Drawing.Primitives.Tests
• System.Memory.Tests
• System.ObjectModel.Tests
• System.Xml.RW.XmlReader.Tests
• Microsoft.Extensions.Options.Tests
• Microsoft.Extensions.Configuration.Tests
• Microsoft.Extensions.Configuration.UserSecrets.Tests
• Microsoft.Extensions.Logging.Testing.Tests
• System.IO.Packaging.Tests
• System.Memory.Data.Tests
• System.Net.Connections.Tests
• System.Net.Primitives.Functional.Tests
• System.Net.Security.Enterprise.Tests
• Microsoft.Extensions.FileProviders.Composite.Tests
• System.Diagnostics.DiagnosticSource.Switches.Tests
• System.Diagnostics.TraceSource.Tests
• Microsoft.Extensions.Configuration.Binder.Tests
• Microsoft.Extensions.Configuration.FileExtensions.Tests
• Microsoft.Extensions.Hosting.Functional.Tests
• System.ComponentModel.Annotations.Tests
• System.Composition.Hosting.Tests
• System.Globalization.Tests
• System.IO.UnmanagedMemoryStream.Tests
• System.Net.WebSockets.Tests
• System.Reflection.CoreCLR.Tests
• System.Reflection.TypeExtensions.Tests
• System.Transactions.Local.Tests
• Microsoft.Win32.Primitives.Tests
• System.ComponentModel.Primitives.Tests
• System.Diagnostics.DiagnosticSource.Tests
• System.Globalization.CalendarsWithConfigSwitch.Tests
• System.IO.FileSystem.DriveInfo.Tests
• System.Net.Http.Enterprise.Tests
• System.Net.Http.Json.Unit.Tests
• System.Net.Http.Unit.Tests
• System.Resources.Reader.Tests
• System.Runtime.CompilerServices.Unsafe.Tests
• System.Runtime.Handles.Tests
• System.Threading.Tasks.Extensions.Tests
• System.Threading.Timer.Tests

Author:	mdh1418
Assignees:	-
Labels:	arch-wasm
Milestone:	-

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[ghost]

Tagging subscribers to this area: @CoffeeFlux
See info in area-owners.md if you want to be subscribed.

Issue Details

---

To see where the wasm library tests stand with AOT compilation with respect to how many pass/fail, what the failures are, how many hang, the 268 test suites were ran locally.

Dropbox paper with findings

---

Suites that did not produce wasm-console.log or testResults.xml (48 suites)

Relevant binlogs

Error: Failed to load method

• System.ComponentModel.Composition.Tests
• System.Diagnostics.Tracing.Tests
• System.Runtime.Loader.DefaultContext.Tests
• System.Runtime.Loader.RefEmitLoadContext.Tests

---

Error: Cannot open image

• System.Diagnostics.FileVersionInfo.Tests

---

FilesToIncludeInFileSystem file not found

• System.Diagnostics.StackTrace.Tests
• System.Reflection.MetadataLoadContext.Tests
• System.Reflection.Tests

---

System.NotSupportedException

• System.Net.NameResolution.Pal.Tests

---

Assertion at mini-wasm.c:117, condition `sig->call_convention != MONO_CALL_VARARG' not met

• System.Runtime.Tests

---

Suites that did not run to completion (did not produce testResults.xml (33 suites)

• Microsoft.CSharp.Tests
• Microsoft.Extensions.Caching.Memory.Tests
• Microsoft.Extensions.DependencyModel.Tests
• Microsoft.Extensions.Hosting.Unit.Tests
• Microsoft.Extensions.Logging.Tests
• Microsoft.VisualBasic.Core.Tests
• System.Console.Tests
• System.Dynamic.Runtime.Tests
• System.IO.IsolatedStorage.Tests
• System.Linq.Parallel.Tests
• System.Linq.Tests
• System.Net.Http.Functional.Tests
• System.Net.Mail.Functional.Tests
• System.Private.Runtime.InteropServices.JavaScript.Tests
• System.Reflection.Context.Tests
• System.Reflection.Emit.ILGeneration.Tests
• System.Reflection.Emit.Tests
• System.Reflection.Metadata.Tests
• System.Runtime.Extensions.Tests
• System.Runtime.InteropServices.Tests
• System.Runtime.Loader.Tests
• System.Runtime.Serialization.Json.ReflectionOnly.Tests
• System.Runtime.Serialization.Xml.ReflectionOnly.Tests
• System.Text.Json.Tests
• System.Threading.Tasks.Dataflow.Tests
• System.Threading.Tasks.Parallel.Tests
• System.Threading.Tasks.Tests
• System.ValueTuple.Tests
• System.Xml.Linq.xNodeBuilder.Tests
• System.Xml.XmlSerializer.ReflectionOnly.Tests
• System.Xml.XmlSerializer.Tests
• System.Xml.Xsl.XslCompiledTransformApi.Tests
• System.Xml.Xsl.XslTransformApi.Tests

---

Suites that failed with 1 or more test failures (descending in number of failures) (103 suites)

• System.Net.Sockets.Tests
• System.Data.Common.Tests
• System.Security.Cryptography.Algorithms.Tests
• System.Numerics.Vectors.Tests
• System.Security.Cryptography.OpenSsl.Tests
• System.Security.Cryptography.Pkcs.Tests
• System.Security.Cryptography.X509Certificates.Tests
• System.Security.Cryptography.Csp.Tests
• System.Linq.Queryable.Tests
• Microsoft.Extensions.DependencyInjection.ExternalContainers.Tests
• System.IO.FileSystem.Tests
• System.Linq.Expressions.Tests
• System.Runtime.Serialization.Xml.Tests
• Microsoft.Extensions.DependencyInjection.Tests
• System.Net.HttpListener.Tests
• System.Security.Cryptography.Xml.Tests
• System.Runtime.Serialization.Json.Tests
• System.Diagnostics.Process.Tests
• System.Security.Cryptography.Encoding.Tests
• System.Composition.Tests
• System.Net.Requests.Tests
• System.Xml.XmlSchema.XmlSchemaValidatorApi.Tests
• System.ComponentModel.TypeConverter.Tests
• System.Net.NetworkInformation.Functional.Tests
• Microsoft.Extensions.FileProviders.Physical.Tests
• System.Threading.Overlapped.Tests
• System.Threading.Channels.Tests
• System.IO.Pipelines.Tests
• System.Net.NameResolution.Functional.Tests
• System.Data.DataSetExtensions.Tests
• System.Runtime.Numerics.Tests
• System.ServiceModel.Syndication.Tests
• System.IO.Compression.Brotli.Tests
• System.Net.WebClient.Tests
• System.Collections.Immutable.Tests
• System.ComponentModel.Composition.Registration.Tests
• System.Reflection.DispatchProxy.Tests
• System.Net.Http.Json.Functional.Tests
• System.Net.Ping.Functional.Tests
• Microsoft.Extensions.Configuration.Json.Tests
• Microsoft.Extensions.Logging.EventSource.Tests
• System.Runtime.InteropServices.RuntimeInformation.Tests
• Microsoft.Extensions.Configuration.Functional.Tests
• System.IO.Compression.Tests
• Common.Tests
• System.Collections.Concurrent.Tests
• System.IO.MemoryMappedFiles.Tests
• System.Composition.Convention.Tests
• System.Text.Encoding.Tests
• System.Threading.Tests
• System.Xml.XmlSchemaSet.Tests
• Microsoft.Extensions.Primitives.Tests
• System.Composition.TypedParts.Tests
• Microsoft.Extensions.Http.Tests
• System.Reflection.Emit.Lightweight.Tests
• Microsoft.Extensions.Configuration.Ini.Tests
• System.IO.Tests
• Microsoft.Extensions.Configuration.Xml.Tests
• System.Security.Cryptography.Primitives.Tests
• System.Collections.NonGeneric.Tests
• System.Collections.Tests
• System.Text.RegularExpressions.Tests
• System.CodeDom.Tests
• System.Drawing.Primitives.Tests
• System.Memory.Tests
• System.ObjectModel.Tests
• System.Xml.RW.XmlReader.Tests
• Microsoft.Extensions.Options.Tests
• Microsoft.Extensions.Configuration.Tests
• Microsoft.Extensions.Configuration.UserSecrets.Tests
• Microsoft.Extensions.Logging.Testing.Tests
• System.IO.Packaging.Tests
• System.Memory.Data.Tests
• System.Net.Connections.Tests
• System.Net.Primitives.Functional.Tests
• System.Net.Security.Enterprise.Tests
• Microsoft.Extensions.FileProviders.Composite.Tests
• System.Diagnostics.DiagnosticSource.Switches.Tests
• System.Diagnostics.TraceSource.Tests
• Microsoft.Extensions.Configuration.Binder.Tests
• Microsoft.Extensions.Configuration.FileExtensions.Tests
• Microsoft.Extensions.Hosting.Functional.Tests
• System.ComponentModel.Annotations.Tests
• System.Composition.Hosting.Tests
• System.Globalization.Tests
• System.IO.UnmanagedMemoryStream.Tests
• System.Net.WebSockets.Tests
• System.Reflection.CoreCLR.Tests
• System.Reflection.TypeExtensions.Tests
• System.Transactions.Local.Tests
• Microsoft.Win32.Primitives.Tests
• System.ComponentModel.Primitives.Tests
• System.Diagnostics.DiagnosticSource.Tests
• System.Globalization.CalendarsWithConfigSwitch.Tests
• System.IO.FileSystem.DriveInfo.Tests
• System.Net.Http.Enterprise.Tests
• System.Net.Http.Json.Unit.Tests
• System.Net.Http.Unit.Tests
• System.Resources.Reader.Tests
• System.Runtime.CompilerServices.Unsafe.Tests
• System.Runtime.Handles.Tests
• System.Threading.Tasks.Extensions.Tests
• System.Threading.Timer.Tests

Author:	mdh1418
Assignees:	-
Labels:	arch-wasm, area-VM-meta-mono, tracking
Milestone:	-

[CoffeeFlux]

It would be nice to compare this with currently known failures with the interpreter, so we can tell what is AOT-specific.

[mdh1418]

I ran all the libraries for Browser wasm here. There were only 5 test failures overall, belonging to two different test suites. However, there were a number of suites that had 0 tests run most likely due to skips.

To compare the failures found from running tests in AOT mode, the test names themselves were compared and out of the tests ran (Not skipped) in interp mode (non AOT), 5275 of the 10974 tests failing in AOT Mode were ran on interp mode. The rest (5699) are likely tests that are skipped on interp mode.

The 5275 tests that failed on AOT yet passing on Interp overall belong to 79 different test suites under sameTests.log in this section

[steveisok]

/cc @lewing @vargaz

[vargaz]

Can the changes be merged, even if turned off by default, so these failures are easier to debug ?

[steveisok]

@vargaz You should be able to run each of these suites now locally. The only thing #48226 does is it shifts the AOT / final link step to helix so that we can spread the work beyond 1 build machine.

[mdh1418]

@vargaz There are a couple of changes on that branch that help run more tests.
4f17c4e without this, many tests are not discovered.
4e7ab85 this one helps avoid some AOT compilation errors with this exception filter. There shouldn't be exception filter issues anymore.

[naricc]

I (@naricc) am working on this issue " System.ArgumentNullException : Value cannot be null. (Parameter 'method')" #46758

[fanyang-mono]

I (@fanyang-mono ) am working on the issue w) System.IO.DirectoryNotFoundException : Could not find a part of the path or System.IO.FileNotFoundException : Could not find file #49232

[radical]

Closing this, as this predates the newer set of aot test issues that are tracked with wasm-aot-test label.