linux ARM64 dotnet7 regression preview 4 to preview 5, 6 and 7

[RobertHenry6bev]

Description

When I attempt to build my application, there is an error coming from crossgen2. I get a different error with dotnet7 preview 5 vs preview 6. Both errors smell like SIGSEGV, and both smell of a corrupted heap. Perhaps other dotnet7 previews have this corruption, but it isn't noticed.

/home/msft/dotnet_all/dotnet7.0/sdk/7.0.100-preview.5.22307.18/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.CrossGen.targets(464,5): error : Process terminated. Access Violation: Attempted to read or write protected memory. This is often an indication that other memory is corrupt. The application will be terminated since this platform does not support throwing an AccessViolationException. [/mnt/msft/dotnet_techempower/aspnet/benchmarks/src/BenchmarksApps/Kestrel/PlatformBenchmarks/PlatformBenchmarks.csproj]

Reproduction Steps

Linux jammy on ARM64, using dotnet7 preview 5 or 6
Running under gdb, the top few stack frame looks like:
Switching to Thread 0xfffff7ff36a0 (LWP 2813)]

0x0000aaaaaae57160 in ILCompiler_TypeSystem_Internal_NativeFormat_TypeHashingAlgorithms__ComputeNameHashCode ()

(gdb) where

#0 0x0000aaaaaae57160 in ILCompiler_TypeSystem_Internal_NativeFormat_TypeHashingAlgorithms__ComputeNameHashCode ()

#1 0x0000aaaaaae5b214 in ILCompiler_TypeSystem_Internal_TypeSystem_InstantiatedType__GetMethod ()

#2 0x0000aaaaaae757e4 in ILCompiler_TypeSystem_Internal_TypeSystem_Ecma_EcmaModule__ResolveMemberReference ()

Expected behavior

no internal errors

Actual behavior

heap corruption

Regression?

No response

Known Workarounds

No response

Configuration

linux (ubuntu jammy 22.04) arm64 dotnet7 preview 4, 5 and 6 (4 OK[sic])
The error's presence and location seem to move around depending on the number of cores available for the job. 3 cores seemed to tickle this problem.

Other information

No response

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[RobertHenry6bev]

area-crossgen2-coreclr

[agocke]

This might be a NativeAOT bug. @RobertHenry6bev any chance you could get a dump file for the crash?

[RobertHenry6bev]

On ARM64 Linux ubuntu 22.04 with dotnet7 preview5, I set environment variables
COMPlus_EnableCrashReport=1
COMPlus_DbgEnableMiniDump=1
COMPlus_CreateDumpDiagnostics=1

I set ulimit -c unlimited
I set kernel config /proc/sys/kernel/core_pattern to /tmp/cores/core.%e.%p.%h.%t
I ran my build script to build my dotnet world.
I got 4 messages like:
/home/robhenry/dotnet_all/dotnet7.0/sdk/7.0.100-preview.5.22307.18/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.CrossGen.targets(464,5): error : Process terminated. Access Violation: Attempted to read or write protected memory. This is often an indication that other memory is corrupt. The application will be terminated since this platform does not support throwing an AccessViolationException. [/mnt/robhenry/dotnet_techempower/aspnet/benchmarks/src/BenchmarksApps/Kestrel/PlatformBenchmarks/PlatformBenchmarks.csproj]
I got 4 core dump files.

I am unable to attach a zip of the corefile, since that is too large. The zip has been transmitted to agocke out-of-band.

[RobertHenry6bev]

I'm trying to build and run a dotnet runtime using gcc with the -fsanitize=thread (tsan), but I'm getting dynamic linking failures preventing me from making more progress

[MichalStrehovsky]

Didn't realize we have this bug. Copying from #72831 that I'll close as a dup:

Very easy to hit on a Raspberry Pi 4 on AArch64 Ubuntu 22.04. We also have a customer report for hitting this on a beefier Ampere device (there it requires running as /usr/bin/taskset --all-tasks --cpu-list 1,2,3 to limit the number of CPUs). Probably any ARM64 device would hit this.

The repro I just tried on Ubuntu:

curl -sSL https://dot.net/v1/dotnet-install.sh | bash /dev/stdin --version 7.0.100-rc.1.22375.3 --install-dir ~/netrc/
~/netrc/dotnet new webapi -o webapi
cd webapi
cat /dev/tty > nuget.config
<configuration>
  <packageSources>
    <add key="dotnet7" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet7/nuget/v3/index.json" />
  </packageSources>
</configuration>
^Z
~/netrc/dotnet publish -r linux-arm64 -c Release /p:PublishTrimmed=true /p:PublishReadyToRun=true

The stack traces are all over the place:

error : Error: Object reference not set to an instance of an object. [/home/michal/webapi/webapi.csproj]
error : System.NullReferenceException: Object reference not set to an instance of an object. [/home/michal/webapi/webapi.csproj]
error :    at System.Threading.Lock.ReleaseContended() + 0x50 [/home/michal/webapi/webapi.csproj]
error :    at System.Threading.Lock.Release() + 0x70 [/home/michal/webapi/webapi.csproj]
error :    at System.Collections.Concurrent.ConcurrentDictionary`2.TryAddInternal(TKey, Nullable`1, TValue, Boolean, Boolean, TValue&) + 0x348 [/home/michal/webapi/webapi.csproj]
error :    at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey, Func`2) + 0xbc [/home/michal/webapi/webapi.csproj]
error :    at ILCompiler.DependencyAnalysis.ReadyToRun.CopiedMetadataBlobNode.WriteMethodTableRvas(NodeFactory, ObjectDataBuilder&, BlobReader&) + 0xd0 [/home/michal/webapi/webapi.csproj]
error :    at ILCompiler.DependencyAnalysis.ReadyToRun.CopiedMetadataBlobNode.GetData(NodeFactory, Boolean) + 0x1b0 [/home/michal/webapi/webapi.csproj]
error :    at ILCompiler.DependencyAnalysis.ObjectNode.GetStaticDependencies(NodeFactory) + 0x48 [/home/michal/webapi/webapi.csproj]
error :    at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer`2.GetStaticDependenciesImpl(DependencyNodeCore`1) + 0x48 [/home/michal/webapi/webapi.csproj]
error :    at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer`2.GetStaticDependencies(DependencyNodeCore`1) + 0x40 [/home/michal/webapi/webapi.csproj]
error :    at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer`2.ProcessMarkStack() + 0xd4 [/home/michal/webapi/webapi.csproj]
error :    at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer`2.ComputeMarkedNodes() + 0x4c [/home/michal/webapi/webapi.csproj]
error :    at ILCompiler.ReadyToRunCodegenCompilation.Compile(String) + 0x3c [/home/michal/webapi/webapi.csproj]
error :    at ILCompiler.Program.RunSingleCompilation(Dictionary`2, InstructionSetSupport, String, Dictionary`2, HashSet`1, CompilerTypeSystemContext) + 0x1418 [/home/michal/webapi/webapi.csproj]
error :    at ILCompiler.Program.Run(String[]) + 0xbbc [/home/michal/webapi/webapi.csproj]
error :    at ILCompiler.Program.Main(String[]) + 0x34 [/home/michal/webapi/webapi.csproj]

error :  ---> ILCompiler.CodeGenerationFailedException: Code generation failed for method '[Microsoft.AspNetCore.Http.Abstractions]Microsoft.AspNetCore.Http.PathString.StartsWithSegments(PathString,StringComparison,PathString&)' [/home/michal/webapi/webapi.csproj]
error :  ---> System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. [/home/michal/webapi/webapi.csproj]
error :    at System.ThrowHelper.ThrowArgumentOutOfRangeException() + 0x28 [/home/michal/webapi/webapi.csproj]
error :    at System.Text.Encoding.GetByteCountWithFallback(Char*, Int32, Int32) + 0x50 [/home/michal/webapi/webapi.csproj]
error :    at System.Text.UTF8Encoding.GetByteCount(String) + 0x84 [/home/michal/webapi/webapi.csproj]
error :    at Internal.JitInterface.CorInfoImpl.StringToUTF8(String) + 0x30 [/home/michal/webapi/webapi.csproj]
error :    at Internal.JitInterface.CorInfoImpl.getMethodNameFromMetadata(CORINFO_METHOD_STRUCT_*, Byte**, Byte**, Byte**) + 0x140 [/home/michal/webapi/webapi.csproj]
error :    at Internal.JitInterface.CorInfoImpl._getMethodNameFromMetadata(IntPtr, IntPtr*, CORINFO_METHOD_STRUCT_*, Byte**, Byte**, Byte**) + 0x5c [/home/michal/webapi/webapi.csproj]

I have a crashdump that I was looking at that has:

[0x3]   crossgen2!SystemNative_Abort + 0x8   
[0x4]   crossgen2!S_P_CoreLib_System_RuntimeExceptionHelpers__FailFast_1 + 0xb8   
[0x5]   crossgen2!S_P_CoreLib_System_RuntimeExceptionHelpers__GetRuntimeException + 0x33c   
[0x6]   crossgen2!S_P_CoreLib_System_Runtime_EH__GetClasslibException + 0x38   
[0x7]   crossgen2!RhThrowHwEx + 0xd8   
[0x8]   crossgen2!RhpThrowHwEx + 0x88   
[0x9]   crossgen2!System_Collections_Concurrent_System_Collections_Concurrent_ConcurrentDictionary_2<System___Canon__Bool>__TryGetValueInternal + 0x44   
[0xa]   crossgen2!System_Collections_Concurrent_System_Collections_Concurrent_ConcurrentDictionary_2<System___Canon__Bool>__GetOrAdd + 0x78   
[0xb]   crossgen2!ILCompiler_ReadyToRun_ILCompiler_DependencyAnalysis_ReadyToRun_ModuleTokenResolver__AddModuleTokenForType + 0x13c   
[0xc]   crossgen2!ILCompiler_ReadyToRun_ILCompiler_DependencyAnalysis_ReadyToRun_ModuleTokenResolver_TokenResolverProvider__GetTypeFromReference + 0x90   
[0xd]   crossgen2!System_Reflection_Metadata_System_Reflection_Metadata_Ecma335_SignatureDecoder_2<System___Canon__System___Canon>__DecodeTypeHandle + 0x124   
[0xe]   crossgen2!System_Reflection_Metadata_System_Reflection_Metadata_Ecma335_SignatureDecoder_2<System___Canon__System___Canon>__DecodeType_0 + 0x348   
[0xf]   crossgen2!System_Reflection_Metadata_System_Reflection_Metadata_Ecma335_SignatureDecoder_2<System___Canon__System___Canon>__DecodeType_0 + 0x2a8   
[0x10]   crossgen2!ILCompiler_ReadyToRun_ILCompiler_DependencyAnalysis_ReadyToRun_ModuleTokenResolver__AddModuleTokenForType + 0x124   
[0x11]   crossgen2!ILCompiler_ReadyToRun_ILCompiler_DependencyAnalysis_ReadyToRun_ModuleTokenResolver__AddModuleTokenForMethod + 0x21c   
[0x12]   crossgen2!ILCompiler_ReadyToRun_Internal_JitInterface_CorInfoImpl__resolveToken + 0x1a8   
[0x13]   crossgen2!ILCompiler_ReadyToRun_Internal_JitInterface_CorInfoImpl___resolveToken + 0x3c   

We've seen a bunch of apparent corruptions around ConcurrentDictionary even before the NativeAOT switchover: see #69192. But I'm not able to repro this with crossgen2 without NativeAOT anymore. So it's either a NativeAOT bug or the bug doesn't repro without regions (we still run GC in a different configuration than the rest of the shipping product unfortunately).

[EgorBo]

I hit a gc assert when I tried to enable Frozen Segments for CoreCLR #49576 (comment) (doesn't repro when I disable gc regions)

can it be related since NativeAOT also uses frozen segments?

[MichalStrehovsky]

We couldn't enable regions yet in NativeAOT because of some bug. Possibly the bug you found. So we don't run with region in NativeAOT yet. We have a 7.0 issue to get them running.

[EgorBo]

We couldn't enable regions yet in NativeAOT because of some bug. Possibly the bug you found. So we don't run with region in NativeAOT yet. We have a 7.0 issue to get them running.

Filed #73110. There is another bug I've just found - it seems when you register a frozen segment you should not set ibAllocated to the size of the segment you requested (not sure my change is correct though) and update it as you allocate objects on that heap

[RobertHenry6bev]

This issue is still here with yesterday's preview release: 7.0.100-preview.7.22377.5
It is still blocking forward progress.

[JulieLeeMSFT]

@MichalStrehovsky, is your team working on the bug fix for this issue? Who is the owner of this issue? It would be good to have an assignee.

@RobertHenry6bev, as I explained in Teams chat, we try to resolve blocking-release issues before GA release.

[MichalStrehovsky]

@MichalStrehovsky, is your team working on the bug fix for this issue? Who is the owner of this issue? It would be good to have an assignee.

I raised it to Jeff today that we need an owner. There is another crossgen2 crash (without NativeAOT this time) on Windows ARM64: #71702. It's not clear if this is codegen, GC, VM, or what, or even whether they are related. I looked at a crashdump - it is a corruption.

[JulieLeeMSFT]

cc @mangod9.

[mangod9]

It is still blocking forward progress.

@RobertHenry6bev just checking whether you are able to get it to publish from an x64 host, as in you can always publish for lin-arm64 from a x64 sdk (from any OS)?

[RobertHenry6bev]

I have not tried this suggestion. Given the kinds and locations of machines we run this workload on, everything needs to work out of the box.

…

On Thu, Aug 11, 2022 at 4:41 PM Manish Godse ***@***.***> wrote: It is still blocking forward progress. @RobertHenry6bev <https://github.com/RobertHenry6bev> just checking whether you are able to get it to publish from an x64 host, as in you can always publish for lin-arm64 from a x64 sdk (from any OS)? — Reply to this email directly, view it on GitHub <#72645 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABBLLY6PCAU7KN2FORS6UV3VYWFT3ANCNFSM54JESG7Q> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[mangod9]

I am able to repro it too, this stack consistently seems to be showing up on recent builds:

    frame #7: 0x0000aad846a37c44 crossgen2`RhpThrowHwEx at ExceptionHandling.S:246
    frame #8: 0x0000aad846c42f34 crossgen2`ILCompiler_TypeSystem_Internal_TypeSystem_InstantiatedType__GetMethod(this=0x0000ff0e5afb14d8, name=<unavailable>, signature=<unavailable>, substitution=<unavailable>) at InstantiatedType.cs:161
    frame #9: 0x0000aad846c42f34 crossgen2`ILCompiler_TypeSystem_Internal_TypeSystem_InstantiatedType__GetMethod(this=0x0000000000000000, name=<unavailable>, signature=<unavailable>, substitution=<unavailable>) at InstantiatedType.cs:161
    frame #10: 0x0000aad846c5a878 crossgen2`ILCompiler_TypeSystem_Internal_TypeSystem_Ecma_EcmaModule__ResolveMemberReference(this=<unavailable>, handle=<unavailable>) at EcmaModule.cs:483
    frame #11: 0x0000aad846c650dc crossgen2`ILCompiler_TypeSystem_Internal_TypeSystem_Ecma_EcmaModule_EcmaObjectLookupHashtable__CreateValueFromKey(this=<unavailable>, handle=<unavailable>) at EcmaModule.cs:118
    frame #12: 0x0000aad846d9e0d0 crossgen2`ILCompiler_TypeSystem_Internal_TypeSystem_LockFreeReaderHashtable_2<System_Reflection_Metadata_System_Reflection_Metadata_EntityHandle__System___Canon>__CreateValueAndEnsureValueIsInTable(this=0x0000ff0e5af64dd0, key=<unavailable>) at LockFreeReaderHashtable.cs:559
    frame #13: 0x0000aad846c5a12c crossgen2`ILCompiler_TypeSystem_Internal_TypeSystem_Ecma_EcmaModule__GetObject(this=<unavailable>, handle=<unavailable>, notFoundBehavior=Throw) at EcmaModule.cs:378
    frame #14: 0x0000aad846cc8608 crossgen2`ILCompiler_ReadyToRun_Internal_JitInterface_CorInfoImpl__ResolveTokenInScope(methodIL=<unavailable>, typeOrMethodContext=<unavailable>, token=<unavailable>) at CorInfoImpl.cs:1585
    frame #15: 0x0000aad846cc89e4 crossgen2`ILCompiler_ReadyToRun_Internal_JitInterface_CorInfoImpl__resolveToken(this=0x0000ff0d5afa9e60, pResolvedToken=0x0000ff0c3eb5bc80) at CorInfoImpl.cs:1656
    frame #16: 0x0000aad846cbe5ac crossgen2`ILCompiler_ReadyToRun_Internal_JitInterface_CorInfoImpl___resolveToken(thisHandle=<unavailable>, ppException=<unavailable>, pResolvedToken=<unavailable>) at CorInfoBase.cs:445```

[MichalStrehovsky]

I can repro this reliable crash (reliably crashes in random spots) on ARM64 Windows as well, which is actually a good news because debugging on Windows is much better than on Linux (can do time travel debugging, etc.).

I'm submitting #74211 to not compile crossgen2 with NativeAOT on arm64. I'll remove the blocking release label from this one. We have a bunch of other stress bugs on CoreCLR-hosted crossgen2 but it crashes a lot less often without NativeAOT and might be actually usable. I'd like to port that to RC1.

[VSadov]

We have a bunch of other stress bugs on CoreCLR-hosted crossgen2 but it crashes a lot less often

Do we have random crashes on CoreClr+arm64 as well?

[mangod9]

We have a bunch of other stress bugs on CoreCLR-hosted crossgen2 but it crashes a lot less often

Do we have random crashes on CoreClr+arm64 as well?

Yeah we have seen regular crossgen2 crashes on arm64, but unfortunately dumps are not being captured.

[a74nh]

I'm seeing this failure on latest HEAD of runtime building on Arm64 Linux.

I see that it fixes when applying #74211, but that PR was removed with #74617

How should I be doing my Arm64 Linux builds?

(edit: fix link)

[kunalspathak]

I'm seeing this failure on latest HEAD of runtime building on Arm64 Linux.

What are the errors?

[AntonLapounov]

Yes, you can hit an intermittent failure building the repo like the one below:

  ILCompiler.RyuJit -> /runtime/artifacts/bin/ILCompiler.RyuJit/arm64/Release/ILCompiler.RyuJit.dll
  ILCompiler -> /runtime/artifacts/bin/coreclr/Linux.arm64.Release/ilc/ilc.dll
  Optimizing assemblies for size. This process might take a while.
/runtime/.dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.CrossGen.targets(465,5): error : Process terminated. Access Violation: Attempted to read or write protected memory. This is often an indication that other memory is corrupt. The application will be terminated since this platform does not support throwing an AccessViolationException. [/runtime/src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj]
/runtime/.dotnet/sdk/7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.CrossGen.targets(353,5): error NETSDK1096: Optimizing assemblies for performance failed. You can either exclude the failing assemblies from being optimized, or set the PublishReadyToRun property to false. [/runtime/src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj]

The issue is that we are using the non-fixed crossgen2 binary from the Preview 7 NuGet package instead of the live one, which is fixed.

[a74nh]

Yes, you can hit an intermittent failure building the repo like the one below

That's the one I'm hitting, but it's every time (HiSilicon D05 with Ubuntu 18.04)

The issue is that we are using the non-fixed crossgen2 binary from the Preview 7 NuGet package instead of the live one, which is fixed.

Is there an easy workaround to force the build to use the live version?
Otherwise should we just be manually reverting #74617 for now?

Do you know roughly when this will be fixed properly?

[AntonLapounov]

Otherwise should we just be manually reverting #74617 for now?

Either that or disable using server GC (DOTNET_gcServer=0).

[VSadov]

disable using server GC (DOTNET_gcServer=0)

I am not sure that will work. The choice of GC mode is a compile-time thing in NativeAOT.

[mangod9]

Guess we should revert #74617 till we update the repo to use RC1.

[VSadov]

I do not think reverting back and forth the way crossgen2 is published has any effect on the crossgen that is actually in use, which is from Prevew 7 Nuget package.

Replacing the binaries in the nuget cache with a locally built copy might work, but is a fragile solution.
I can't think of a better workaround.

(I am also annoyed by the failures, my solution is basically try again, sometimes a few times)

[AntonLapounov]

disable using server GC (DOTNET_gcServer=0)

I am not sure that will work. The choice of GC mode is a compile-time thing in NativeAOT.

Yes, it works:

runtime/src/coreclr/nativeaot/Runtime/gcrhenv.cpp

Line 175 in 1d27c3d

g_heap_type = (g_pRhConfig->GetgcServer() && PalGetProcessCpuCount() > 1) ? GC_HEAP_SVR : GC_HEAP_WKS;

[AntonLapounov]

Guess we should revert #74617 till we update the repo to use RC1.

Note that we actually need to check whether the host architecture is ARM64 instead of checking whether the target architecture is ARM64. If you target ARM64 on an x64 machine, you don't need that workaround.

[VSadov]

Ah, I see, SVR build is a superset of WKS, so it is possible to force crossgen2 into WKS mode.

This would be a nice workaround!

[AntonLapounov]

Yep, and you can disable it in the config file just for this binary (e.g., /home/username/.nuget/packages/microsoft.netcore.app.crossgen2.linux-arm64/7.0.0-preview.7.22375.6/tools/crossgen2.runtimeconfig.json).

[MichalStrehovsky]

Otherwise should we just be manually reverting #74617 for now?

I don't think reverting that one has a connection to the crossgen2 crash while crossgenning ILC that Anton described (#72645 (comment)). That pull request doesn't affect the crossgen that is in use to crossgen ILC. It affects future crossgen in a future SDK.

I've submitted #74856 to work around.

Yep, and you can disable it in the config file just for this binary (e.g., /home/username/.nuget/packages/microsoft.netcore.app.crossgen2.linux-arm64/7.0.0-preview.7.22375.6/tools/crossgen2.runtimeconfig.json).

NativeAOT doesn't read runtimeconfig.json so this is likely just placebo effect. We were generating unused runtimeconfig and deps json files with NativeAOT in the past but that has been fixed.

[AntonLapounov]

NativeAOT doesn't read runtimeconfig.json so this is likely just placebo effect.

Thank you for the corrections. I have never actually tried that, just assumed that if we produce a file, it should probably work. :) Setting DOTNET_gcServer=0 certainly works for me.