Revert Revert "Copy published crossgen2 in artifacts/tests (#80154)" #106965

am11 · 2024-08-26T13:49:50Z

am11 · 2024-08-26T13:51:46Z

cc @jakobbotsch second commit has the three fixes. Lets see if crossgen2-outerloop and superpmi-collect legs agree.

jakobbotsch · 2024-08-26T13:56:37Z

Can you run https://github.com/dotnet/runtime/blob/main/src/coreclr/tools/Common/JitInterface/ThunkGenerator/gen.sh to update the JIT-EE GUID? Then I will run internal superpmi-collect to check.

jakobbotsch · 2024-08-26T14:01:05Z

/azp run runtime-coreclr crossgen2 outerloop

azure-pipelines · 2024-08-26T14:01:18Z

Azure Pipelines successfully started running 1 pipeline(s).

jakobbotsch · 2024-08-26T14:01:45Z

/azp run runtime-coreclr outerloop

azure-pipelines · 2024-08-26T14:02:00Z

Azure Pipelines successfully started running 1 pipeline(s).

src/tests/Common/Directory.Build.targets

jakobbotsch · 2024-08-26T14:15:10Z

/azp run Fuzzlyn

azure-pipelines · 2024-08-26T14:15:22Z

Azure Pipelines successfully started running 1 pipeline(s).

jakobbotsch · 2024-08-27T08:40:43Z

Fuzzlyn run looks fine (failures are actual product bugs).

Outerloop run still has the #106948 #106949 failures. I don't see them on the rolling run, so they do seem related to or exposed by this PR.

The "Test crossgen2-comparison build X Y" job failures still look related. Example log: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/788469/logs/199

superpmi-collect failed internally with the same error as before. I'm trying to get a public test version of that pipeline set up in #106960

superpmi-collect full log

2024-08-26T15:07:42.7579217Z ##[section]Starting: Create Core_Root
2024-08-26T15:07:42.7582919Z ==============================================================================
2024-08-26T15:07:42.7583014Z Task         : Command line
2024-08-26T15:07:42.7583063Z Description  : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows
2024-08-26T15:07:42.7583148Z Version      : 2.237.1
2024-08-26T15:07:42.7583197Z Author       : Microsoft Corporation
2024-08-26T15:07:42.7583252Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/command-line
2024-08-26T15:07:42.7583335Z ==============================================================================
2024-08-26T15:07:42.8793067Z Generating script.
2024-08-26T15:07:42.8803748Z Script contents:
2024-08-26T15:07:42.8805790Z /mnt/vss/_work/1/s/src/tests/build.sh checked arm64 generatelayoutonly  /p:LibrariesConfiguration=Release
2024-08-26T15:07:42.8806462Z ========================== Starting Command Output ===========================
2024-08-26T15:07:42.8827337Z [command]/usr/bin/bash --noprofile --norc /mnt/vss/_work/_temp/d76edee6-402c-434b-85c3-bdade8143a67.sh
2024-08-26T15:07:43.1603516Z __OutputRid: linux-arm64
2024-08-26T15:07:43.1654746Z Building Tests...
2024-08-26T15:07:43.1725438Z __TargetOS: linux
2024-08-26T15:07:43.1725718Z __TargetArch: arm64
2024-08-26T15:07:43.1728575Z __BuildType: Checked
2024-08-26T15:07:43.1728895Z __TestIntermediatesDir: /mnt/vss/_work/1/s/artifacts/tests/coreclr/obj/linux.arm64.Checked
2024-08-26T15:07:43.1729270Z __NativeTestIntermediatesDir: /mnt/vss/_work/1/s/artifacts/tests/coreclr/obj/linux.arm64.Checked/Native
2024-08-26T15:07:43.1729710Z __ManagedTestIntermediatesDir: /mnt/vss/_work/1/s/artifacts/tests/coreclr/obj/linux.arm64.Checked/Managed
2024-08-26T15:07:43.1730381Z Creating TestBinDir: /mnt/vss/_work/1/s/artifacts/tests/coreclr/linux.arm64.Checked
2024-08-26T15:07:43.1749080Z Creating LogsDir: /mnt/vss/_work/1/s/artifacts/log
2024-08-26T15:07:43.1768502Z Creating MsbuildDebugLogsDir: /mnt/vss/_work/1/s/artifacts/log/MsbuildDebugLogs
2024-08-26T15:07:43.1792348Z Building tests via "/mnt/vss/_work/1/s/eng/common/msbuild.sh"  --warnAsError false /mnt/vss/_work/1/s/src/tests/build.proj /t:TestBuild /p:TargetArchitecture=arm64 /p:Configuration=Checked /p:TargetOS=linux /nodeReuse:false    /maxcpucount "/flp:Verbosity=normal;LogFile=/mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.log" "/flp1:WarningsOnly;LogFile=/mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.wrn" "/flp2:ErrorsOnly;LogFile=/mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.err" "/bl:/mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.binlog" /p:NUMBER_OF_PROCESSORS=4 /p:LibrariesConfiguration=Release
2024-08-26T15:07:43.3538949Z Downloading 'https://dotnet.microsoft.com/download/dotnet/scripts/v1/dotnet-install.sh'
2024-08-26T15:07:43.5458709Z   Attempting to install 'sdk v9.0.100-preview.7.24407.12' from public_location.
2024-08-26T15:07:43.8359071Z dotnet-install: Attempting to download using primary link https://dotnetcli.azureedge.net/dotnet/Sdk/9.0.100-preview.7.24407.12/dotnet-sdk-9.0.100-preview.7.24407.12-linux-x64.tar.gz
2024-08-26T15:07:46.2110330Z dotnet-install: Remote file https://dotnetcli.azureedge.net/dotnet/Sdk/9.0.100-preview.7.24407.12/dotnet-sdk-9.0.100-preview.7.24407.12-linux-x64.tar.gz size is 205150545 bytes.
2024-08-26T15:07:46.2113258Z dotnet-install: Extracting archive from https://dotnetcli.azureedge.net/dotnet/Sdk/9.0.100-preview.7.24407.12/dotnet-sdk-9.0.100-preview.7.24407.12-linux-x64.tar.gz
2024-08-26T15:07:52.6982427Z dotnet-install: Downloaded file size is 205150545 bytes.
2024-08-26T15:07:52.6982948Z dotnet-install: The remote and local file sizes are equal.
2024-08-26T15:07:53.1109431Z dotnet-install: Installed version is 9.0.100-preview.7.24407.12
2024-08-26T15:07:53.1185394Z dotnet-install: Adding to current process PATH: `/mnt/vss/_work/1/s/.dotnet`. Note: This change will be visible only when sourcing script.
2024-08-26T15:07:53.1186037Z dotnet-install: Note that the script does not resolve dependencies during installation.
2024-08-26T15:07:53.1186902Z dotnet-install: To check the list of dependencies, go to https://learn.microsoft.com/dotnet/core/install, select your operating system and check the "Dependencies" section.
2024-08-26T15:07:53.1187222Z dotnet-install: Installation finished successfully.
2024-08-26T15:07:53.9185145Z 
2024-08-26T15:07:55.3341936Z   [15:07:55.33] Restoring all packages...
2024-08-26T15:07:55.5964853Z   
2024-08-26T15:07:56.2277389Z     Determining projects to restore...
2024-08-26T15:08:02.8336187Z     Restored /mnt/vss/_work/1/s/src/tests/Common/test_dependencies_fs/test_dependencies.fsproj (in 5.95 sec).
2024-08-26T15:08:03.1947879Z   
2024-08-26T15:08:03.8483224Z     Determining projects to restore...
2024-08-26T15:08:05.0716183Z     Restored /mnt/vss/_work/1/s/src/tests/Common/test_dependencies/test_dependencies.csproj (in 646 ms).
2024-08-26T15:08:05.3844733Z   
2024-08-26T15:08:06.0119968Z     Determining projects to restore...
2024-08-26T15:08:07.6024028Z     Restored /mnt/vss/_work/1/s/src/tests/Common/CoreCLRTestLibrary/CoreCLRTestLibrary.csproj (in 1.01 sec).
2024-08-26T15:08:07.9072114Z   
2024-08-26T15:08:08.5017637Z     Determining projects to restore...
2024-08-26T15:08:09.2113281Z     Restored /mnt/vss/_work/1/s/src/tests/Common/GenerateHWIntrinsicTests/GenerateHWIntrinsicTests_Arm.csproj (in 141 ms).
2024-08-26T15:08:09.5104624Z   
2024-08-26T15:08:10.1003679Z     Determining projects to restore...
2024-08-26T15:08:10.8072973Z     Restored /mnt/vss/_work/1/s/src/tests/Common/GenerateHWIntrinsicTests/GenerateHWIntrinsicTests_General.csproj (in 140 ms).
2024-08-26T15:08:11.1172434Z   
2024-08-26T15:08:11.7144653Z     Determining projects to restore...
2024-08-26T15:08:12.4401917Z     Restored /mnt/vss/_work/1/s/src/tests/Common/GenerateHWIntrinsicTests/GenerateHWIntrinsicTests_X86.csproj (in 137 ms).
2024-08-26T15:08:12.7256903Z   
2024-08-26T15:08:13.3232739Z     Determining projects to restore...
2024-08-26T15:08:14.0382600Z     Restored /mnt/vss/_work/1/s/src/tests/Common/XUnitLogChecker/XUnitLogChecker.csproj (in 138 ms).
2024-08-26T15:08:14.3326861Z   
2024-08-26T15:08:14.9241455Z     Determining projects to restore...
2024-08-26T15:08:18.6261668Z     Restored /mnt/vss/_work/1/s/src/tests/Common/XUnitWrapperGenerator/XUnitWrapperGenerator.csproj (in 3.13 sec).
2024-08-26T15:08:18.9458302Z   
2024-08-26T15:08:19.5546027Z     Determining projects to restore...
2024-08-26T15:08:20.2806869Z     Restored /mnt/vss/_work/1/s/src/tests/Common/XUnitWrapperLibrary/XUnitWrapperLibrary.csproj (in 135 ms).
2024-08-26T15:08:20.5749889Z   
2024-08-26T15:08:21.1573576Z     Determining projects to restore...
2024-08-26T15:08:22.5306653Z     Restored /mnt/vss/_work/1/s/src/tests/Common/XHarnessRunnerLibrary/XHarnessRunnerLibrary.csproj (in 710 ms).
2024-08-26T15:08:22.5352056Z     1 of 2 projects are up-to-date for restore.
2024-08-26T15:08:22.8700126Z   
2024-08-26T15:08:23.5122215Z     Determining projects to restore...
2024-08-26T15:08:25.3625703Z     Restored /mnt/vss/_work/1/s/src/tests/Common/external/external.csproj (in 1.25 sec).
2024-08-26T15:08:25.6653569Z   
2024-08-26T15:08:28.5176711Z     Determining projects to restore...
2024-08-26T15:08:29.7457355Z     Restored /mnt/vss/_work/1/s/src/tests/Common/ilasm/ilasm.ilproj (in 634 ms).
2024-08-26T15:08:29.8036360Z   [15:08:29.80] Restoring all packages...Done.
2024-08-26T15:08:29.8046327Z   Building Targeting Pack
2024-08-26T15:08:30.4660518Z   external -> 
2024-08-26T15:08:31.0923175Z   Determining projects to restore...
2024-08-26T15:08:33.3667587Z   Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/ILCompiler.ReadyToRun/ILCompiler.ReadyToRun.csproj (in 1.08 sec).
2024-08-26T15:08:33.3709757Z   Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj (in 1.08 sec).
2024-08-26T15:08:33.3769852Z   Restored /mnt/vss/_work/1/s/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Crossgen2.sfxproj (in 1.12 sec).
2024-08-26T15:08:33.4599576Z   Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/ILCompiler.Diagnostics/ILCompiler.Diagnostics.csproj (in 8 ms).
2024-08-26T15:08:34.0627181Z   Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/ILCompiler.DependencyAnalysisFramework/ILCompiler.DependencyAnalysisFramework.csproj (in 662 ms).
2024-08-26T15:08:34.0637661Z   Restored /mnt/vss/_work/1/s/src/coreclr/tools/aot/ILCompiler.TypeSystem/ILCompiler.TypeSystem.csproj (in 1.8 sec).
2024-08-26T15:08:34.2090103Z   Restored /mnt/vss/_work/1/s/src/tools/illink/src/linker/ref/Mono.Linker.csproj (in 822 ms).
2024-08-26T15:08:34.3910518Z   Restored /mnt/vss/_work/1/s/src/tools/illink/src/ILLink.RoslynAnalyzer/ILLink.RoslynAnalyzer.csproj (in 334 ms).
2024-08-26T15:08:34.3970413Z   Restored /mnt/vss/_work/1/s/src/tools/illink/src/ILLink.CodeFix/ILLink.CodeFixProvider.csproj (in 187 ms).
2024-08-26T15:08:36.0203374Z   Restored /mnt/vss/_work/1/s/src/libraries/System.Runtime.InteropServices/gen/DownlevelLibraryImportGenerator/DownlevelLibraryImportGenerator.csproj (in 1.61 sec).
2024-08-26T15:08:36.0209690Z   Restored /mnt/vss/_work/1/s/src/tools/illink/src/linker/Mono.Linker.csproj (in 2.62 sec).
2024-08-26T15:08:36.0459693Z   Restored /mnt/vss/_work/1/s/src/libraries/System.Runtime.InteropServices/gen/Microsoft.Interop.SourceGeneration/Microsoft.Interop.SourceGeneration.csproj (in 1.65 sec).
2024-08-26T15:08:40.2405547Z   Restored /mnt/vss/_work/1/s/src/tools/illink/src/ILLink.Tasks/ILLink.Tasks.csproj (in 6.2 sec).
2024-08-26T15:08:49.6915161Z   ILLink.RoslynAnalyzer -> /mnt/vss/_work/1/s/artifacts/bin/ILLink.RoslynAnalyzer/Release/netstandard2.0/ILLink.RoslynAnalyzer.dll
2024-08-26T15:08:50.6188398Z   ILLink.CodeFixProvider -> /mnt/vss/_work/1/s/artifacts/bin/ILLink.CodeFixProvider/Release/netstandard2.0/ILLink.CodeFixProvider.dll
2024-08-26T15:08:52.3396144Z   Mono.Linker -> /mnt/vss/_work/1/s/artifacts/bin/Mono.Linker/ref/Release/net9.0/illink.dll
2024-08-26T15:09:03.5545096Z   Mono.Linker -> /mnt/vss/_work/1/s/artifacts/bin/Mono.Linker/Release/net9.0/linux-arm64/illink.dll
2024-08-26T15:09:04.7397572Z   ILLink.Tasks -> /mnt/vss/_work/1/s/artifacts/bin/ILLink.Tasks/Release/net9.0/linux-arm64/ILLink.Tasks.dll
2024-08-26T15:09:15.0413140Z   Mono.Linker -> /mnt/vss/_work/1/s/artifacts/bin/Mono.Linker/Release/net9.0/illink.dll
2024-08-26T15:09:15.5428508Z   ILLink.Tasks -> /mnt/vss/_work/1/s/artifacts/bin/ILLink.Tasks/Release/net9.0/ILLink.Tasks.dll
2024-08-26T15:09:17.6100414Z   ILCompiler.DependencyAnalysisFramework -> /mnt/vss/_work/1/s/artifacts/bin/ILCompiler.DependencyAnalysisFramework/arm64/Checked/ILCompiler.DependencyAnalysisFramework.dll
2024-08-26T15:09:20.1627795Z   ILCompiler.TypeSystem -> /mnt/vss/_work/1/s/artifacts/bin/ILCompiler.TypeSystem/arm64/Checked/ILCompiler.TypeSystem.dll
2024-08-26T15:09:20.8547055Z   ILCompiler.Diagnostics -> /mnt/vss/_work/1/s/artifacts/bin/ILCompiler.Diagnostics/arm64/Checked/ILCompiler.Diagnostics.dll
2024-08-26T15:09:22.8673023Z   ILCompiler.ReadyToRun -> /mnt/vss/_work/1/s/artifacts/bin/ILCompiler.ReadyToRun/arm64/Checked/ILCompiler.ReadyToRun.dll
2024-08-26T15:09:24.9410932Z   crossgen2_publish -> /mnt/vss/_work/1/s/artifacts/bin/crossgen2_publish/arm64/Checked/crossgen2.dll
2024-08-26T15:09:34.3438288Z   Generating native code
2024-08-26T15:09:43.3319288Z   Unhandled exception: System.DllNotFoundException: Unable to load shared library 'clrjit_universal_arm64_x64' or one of its dependencies. In order to help diagnose loading problems, consider using a tool like strace. If you're using glibc, consider setting the LD_DEBUG environment variable: 
2024-08-26T15:09:43.3320904Z   /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/clrjit_universal_arm64_x64.so: cannot open shared object file: No such file or directory
2024-08-26T15:09:43.3321910Z   /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.38' not found (required by /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/libclrjit_universal_arm64_x64.so)
2024-08-26T15:09:43.3322550Z   libclrjit_universal_arm64_x64.so: cannot open shared object file: No such file or directory
2024-08-26T15:09:43.3323249Z   /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/clrjit_universal_arm64_x64: cannot open shared object file: No such file or directory
2024-08-26T15:09:43.3323657Z   /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/libclrjit_universal_arm64_x64: cannot open shared object file: No such file or directory
2024-08-26T15:09:43.3323945Z   
2024-08-26T15:09:43.3526129Z      at System.Runtime.InteropServices.NativeLibrary.LoadLibraryByName(String libraryName, Assembly assembly, Nullable`1 searchPath, Boolean throwOnError)
2024-08-26T15:09:43.3527401Z      at Internal.JitInterface.JitConfigProvider.<>c__DisplayClass5_0.<Initialize>b__0(String libName, Assembly assembly, Nullable`1 searchPath) in /_/src/coreclr/tools/Common/JitInterface/JitConfigProvider.cs:line 48
2024-08-26T15:09:43.3528229Z      at System.Runtime.InteropServices.NativeLibrary.LoadLibraryCallbackStub(String libraryName, Assembly assembly, Boolean hasDllImportSearchPathFlags, UInt32 dllImportSearchPathFlags)
2024-08-26T15:09:43.3528604Z   
2024-08-26T15:09:43.3530523Z      at Internal.JitInterface.CorInfoImpl.jitStartup(IntPtr host)
2024-08-26T15:09:43.3531343Z      at Internal.JitInterface.CorInfoImpl.jitStartup(IntPtr host)
2024-08-26T15:09:43.3531718Z      at Internal.JitInterface.CorInfoImpl.Startup(CORINFO_OS os) in /_/src/coreclr/tools/Common/JitInterface/CorInfoImpl.cs:line 177
2024-08-26T15:09:43.3532138Z      at ILCompiler.RyuJitCompilationBuilder.ToCompilation() in /_/src/coreclr/tools/aot/ILCompiler.RyuJit/Compiler/RyuJitCompilationBuilder.cs:line 127
2024-08-26T15:09:43.3532538Z      at ILCompiler.Program.Run() in /_/src/coreclr/tools/aot/ILCompiler/Program.cs:line 580
2024-08-26T15:09:43.3533156Z      at ILCompiler.ILCompilerRootCommand.<>c__DisplayClass240_0.<.ctor>b__0(ParseResult result) in /_/src/coreclr/tools/aot/ILCompiler/ILCompilerRootCommand.cs:line 293
2024-08-26T15:09:43.3533552Z      at System.CommandLine.Invocation.InvocationPipeline.Invoke(ParseResult parseResult)
2024-08-26T15:09:43.3748124Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/build/Microsoft.NETCore.Native.targets(317,5): error MSB3073: The command ""/mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/ilc" @"/mnt/vss/_work/1/s/artifacts/obj/coreclr/crossgen2_publish/linux.arm64.Checked/native/crossgen2.ilc.rsp"" exited with code 1. [/mnt/vss/_work/1/s/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]
2024-08-26T15:09:43.8123694Z   Determining projects to restore...
2024-08-26T15:09:45.0851704Z   All projects are up-to-date for restore.
2024-08-26T15:09:45.1221142Z 
2024-08-26T15:09:45.1222122Z Build FAILED.
2024-08-26T15:09:45.1222501Z 
2024-08-26T15:09:45.1223997Z /mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/build/Microsoft.NETCore.Native.targets(317,5): error MSB3073: The command ""/mnt/vss/_work/1/s/artifacts/bin/coreclr/linux.arm64.Checked/x64/ilc/ilc" @"/mnt/vss/_work/1/s/artifacts/obj/coreclr/crossgen2_publish/linux.arm64.Checked/native/crossgen2.ilc.rsp"" exited with code 1. [/mnt/vss/_work/1/s/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]
2024-08-26T15:09:45.1224885Z     0 Warning(s)
2024-08-26T15:09:45.1225288Z     1 Error(s)
2024-08-26T15:09:45.1225694Z 
2024-08-26T15:09:45.1226208Z Time Elapsed 00:01:51.20
2024-08-26T15:09:45.2065399Z Build failed with exit code 1. Check errors above.
2024-08-26T15:09:45.2068955Z Failed to build tests. See the build logs:
2024-08-26T15:09:45.2069332Z     /mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.log
2024-08-26T15:09:45.2069654Z     /mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.wrn
2024-08-26T15:09:45.2069947Z     /mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.err
2024-08-26T15:09:45.2070232Z     /mnt/vss/_work/1/s/artifacts/log/TestBuild.linux.arm64.Checked.binlog
2024-08-26T15:09:45.2087847Z 
2024-08-26T15:09:45.2136613Z ##[error]Bash exited with code '1'.
2024-08-26T15:09:45.2151015Z ##[section]Finishing: Create Core_Root

am11 · 2024-08-27T08:44:34Z

I was investigating it in the morning #107019. It's basically paths issue. In comparison job, the "core_root" is repurposed. It's not the same core_root we are using in other places; it's pointer to dlls/ directory instead of core_root which contains corerun/crossgen2 etc.

am11 · 2024-08-27T10:53:55Z

This seems like a product issue in published crossgen2.exe (https://nuget.info/packages/Microsoft.NETCore.App.Crossgen2.win-x64/9.0.0-preview.7.24405.7 tools/crossgen2.exe) which the tests are exposing:

https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-106965-merge-ce81f0ec17e94a889e/CoreMangLib/1/console.5dbc992d.log?helixlogtype=result

Response file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll.rsp
C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\IL-CG2\*.dll
-o:C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll
--targetarch:x64
--targetos:windows
--verify-type-and-field-layout
--method-layout:random
-r:C:\h\w\BAE10A27\p\System.*.dll
-r:C:\h\w\BAE10A27\p\Microsoft.*.dll
-r:C:\h\w\BAE10A27\p\xunit.*.dll
-r:C:\h\w\BAE10A27\p\mscorlib.dll
-r:C:\h\w\BAE10A27\p\netstandard.dll

" "C:\h\w\BAE10A27\p\crossgen2\crossgen2.exe" @"C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll.rsp"   --composite"
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\CoreMangLib.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateCombine1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateCombineImpl.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateEquals1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateGetHashCode1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\DelegateRemove.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\delegateRemoveImpl.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedAdd1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedAdd2.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedCompareExchange1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedCompareExchange5.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedCompareExchange6.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedCompareExchange7.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedDecrement1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedDecrement2.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedExchange1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedExchange5.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedExchange6.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedExchange7.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedIncrement1.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\InterlockedIncrement2.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\TestLibrary.dll
Emitting R2R PE file: C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\XUnitWrapperLibrary.dll

" "dotnet" "C:\h\w\BAE10A27\p\r2rdump\r2rdump.dll" --header --sc --in C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll --out C:\h\w\BAE10A27\w\9A36082A\e\CoreMangLib\CoreMangLib\\composite-r2r.dll.r2rdump --val"
16:55:40.39
"C:\h\w\BAE10A27\p\watchdog.exe" 89 "C:\h\w\BAE10A27\p\corerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false" -p "System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true"  CoreMangLib.dll 

Assert failure(PID 692 [0x000002b4], Thread: 7360 [0x1cc0]): Consistency check failed: Crst Level violation: Can't take level 6 lock CrstUnresolvedClassLock because you already holding level 0 lock CrstNativeImageEagerFixups
FAILED: false

CORECLR! CHECK::Trigger + 0x1F0 (0x00007ffe`774fef80)
CORECLR! CrstBase::IsSafeToTake + 0x38D (0x00007ffe`76e2ca4d)
CORECLR! CrstBase::Enter + 0x1AF (0x00007ffe`76e2c31f)
CORECLR! ClassLoader::LoadTypeHandleForTypeKey_Body + 0x94E (0x00007ffe`76e1372e)
CORECLR! ClassLoader::LoadTypeHandleForTypeKey + 0x346 (0x00007ffe`76e12656)
CORECLR! ClassLoader::LoadTypeDefThrowing + 0x1087 (0x00007ffe`76e12037)
CORECLR! ClassLoader::LoadTypeHandleThrowing + 0x9FD (0x00007ffe`76e1581d)
CORECLR! ClassLoader::LoadTypeHandleThrowIfFailed + 0x937 (0x00007ffe`76e14bd7)
CORECLR! ClassLoader::LoadTypeByNameThrowing + 0x7B1 (0x00007ffe`76e0f541)
CORECLR! CoreLibBinder::LookupClassLocal + 0x383 (0x00007ffe`76d96103)
    File: D:\a\_work\1\s\src\coreclr\vm\crst.cpp:765
    Image: C:\h\w\BAE10A27\p\corerun.exe

App Exit Code: -1073740286
Expected: 100
Actual: -1073740286
END EXECUTION - FAILED

Note that it is not failing on emitting the R2R image with crossgen2, but on usage of the emitted image with r2rdump. I can switch it back to use dotnet crossgen2.dll, but that will defeat the purpose of the PR (to make tests use the same form-factor which is shipped in nupkg).

am11 · 2024-08-27T14:54:14Z

It is unrelated issue, happening in current main as well: https://dev.azure.com/dnceng-public/public/_build/results?buildId=789431&view=results. It turned out that corerun.exe dummy.dll is also throwing this error during the runtime startup..

am11 · 2024-08-27T15:09:12Z

@jakobbotsch, @jkoritzinsky could you please rerun the crossgen2 outerloop leg. I think I have fixed the path issue in the yaml file, but there is only one way to be sure. 😅

jakobbotsch · 2024-08-27T15:28:43Z

/azp run runtime-coreclr crossgen2 outerloop

azure-pipelines · 2024-08-27T15:29:04Z

Azure Pipelines successfully started running 1 pipeline(s).

)" This reverts commit b579962.

am11 · 2024-08-28T11:40:22Z

2024-08-27T22:19:36.6582220Z convert value "System.String" to type "System.Management.Automation.SwitchParameter". Boolean parameters accept only

It was the PowerShell thing. eng/common/msbuild.ps1 has prepareMachine arg (which starts with p). PowerShell has a "parameter aliasing" feature due to which it was mapping cmdlet arg -p:Crossgen2Supported=false to prepareMachine in msbuild.ps1, while our intention was it to be passed to the actual msbuild.exe that the ps1 calls. So the fix is to use /p: instead of -p: (or quote '-p:Crossgen2Supported=false').

am11 · 2024-08-28T12:01:25Z

I hope now this is ready. Lets run the legs again to see it in action. 😎

jkotas · 2024-08-28T14:27:36Z

We can actually use the published variant everywhere, but it comes with extra complexity. e.g. in crossgen2-comparison-build-job.yml, we cross build product (clr+libs subsets) for arm64, then use x64 crossgen2.dll (which was built as part of the arm64 build) to collect the baseline measurements on the host (x64) machine.

These types of scenarios are one of the reasons why I prefer to use non-live AOT compilers to compile tools and achieve building with live AOT compilers by building twice like source build - first build produces live AOT compilers, and second build uses them to build the tools.

jakobbotsch · 2024-08-28T14:31:54Z

I hope now this is ready. Lets run the legs again to see it in action. 😎

I'll run this asap once #106960 is merged. Also there's a Helix outage right now that hopefully should be getting resolved soon...

am11 · 2024-08-28T15:46:14Z

These types of scenarios are one of the reasons why I prefer to use non-live AOT compilers to compile tools and achieve building with live AOT compilers by building twice like source build - first build produces live AOT compilers, and second build uses them to build the tools.

@jkotas, I agree that building twice would be the right approach. However, I think we can completely skip the non-live variant if this alternative hybrid approach is feasible: move the ILCompiler and crossgen2 packaging at the end of the libs subset. It will use the live MSIL variant dotnet ilc.dll and live libs to PublishAot the ILCompiler project and dotnet pack the temporary (stage0) nupkg, then use this stage0 package to publish ILCompiler (again) and crossgen2 projects. Time spent on build and rebuild of those two sfprojs would be observable from current baseline of clr+libs or mono+libs builds.

jkotas · 2024-08-29T05:08:35Z

I understand that you can make it work. We have used live builds in various forms since .NET Framework 1.0 days. Personally, I dislike using the live build as the default build experience for following reasons:

Debug/checked build is significantly slower. If you use live debug or checked build to publish the AOT compiler, the compiler is significantly slower since you pay the perf penalty twice: for the runtime that the AOT compiler runs on and for the runtime that the AOT compiler targets.
Using live build makes troubleshooting bugs harder. If you introduce bug in the core compiler, you are often not able to run the tests because the runtime that the AOT compiler runs on is crashing. You have to either apply workaround to use non-live build so that you can run the tests that typically find the bug in more targeted way, or you have to go through the pain of debugging the AOT compiler.
Using live binaries during the build tends to have problems with antiviruses and other types of locked down environments since you are running unsigned binaries.

jakobbotsch · 2024-08-29T13:17:08Z

/azp run runtime-coreclr crossgen2 outerloop, runtime-coreclr superpmi-collect-test

azure-pipelines · 2024-08-29T13:17:40Z

Azure Pipelines successfully started running 2 pipeline(s).

jakobbotsch · 2024-08-29T13:20:26Z

Is the discussion above related to changes in this PR? Or is it purely about potential future changes?

am11 · 2024-08-29T13:48:33Z

Those are not related to the Tests/Core_Root but rather publishing of nupkgs we ship.

jakobbotsch · 2024-08-30T07:24:33Z

The crossgen2 superpmi-collect failures look related. E.g. https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-106965-merge-cceb9555ca7e45ce80/0/1/console.c6f65cfc.log?helixlogtype=result:

C:\h\w\BA150988\w\B75C0A24\e>C:\python3\python.exe C:\h\w\BA150988\p\superpmi\superpmi.py collect --clean -log_level DEBUG --crossgen2  -assemblies C:\h\w\BA150988\w\B75C0A24\u\binaries -arch x64 -build_type checked -core_root C:\h\w\BA150988\p\superpmi -output_mch_path C:\h\w\BA150988\w\B75C0A24\uploads\libraries.crossgen2.0.windows.x64.checked.mch -log_file C:\h\w\BA150988\w\B75C0A24\uploads\libraries.crossgen2.0.windows.x64.checked.log 
[15:20:14] ================ Logging to C:\h\w\BA150988\w\B75C0A24\uploads\libraries.crossgen2.0.windows.x64.checked.log
`--crossgen2` is specified, but couldn't find C:\h\w\BA150988\p\superpmi\crossgen2\crossgen2.exe. (Is it built?)

C:\h\w\BA150988\w\B75C0A24\e>set _commandExitCode=1 

C:\h\w\BA150988\w\B75C0A24\e>taskkill.exe /f /im corerun.exe 
ERROR: The process "corerun.exe" not found.

C:\h\w\BA150988\w\B75C0A24\e>EXIT /b 1 
['0' END OF WORK ITEM LOG: Command exited with 1]

am11 · 2024-08-30T09:34:23Z

@jakobbotsch crossgen2 publishing from all the superpmi pipelines are disabled but I forgot to revert superpmi.py change which was expecting crossgen2.exe instead of .dll.

jakobbotsch · 2024-08-30T13:18:06Z

/azp run runtime-coreclr superpmi-collect-test

azure-pipelines · 2024-08-30T13:18:31Z

Azure Pipelines successfully started running 1 pipeline(s).

am11 · 2024-08-31T11:03:30Z

Remaining failures are same as the one in main. Opened #107226.

jakobbotsch · 2024-09-02T11:33:36Z

Thanks for fixing the remaining problems.

)" (dotnet#106965) This reverts commit b579962.

dotnet-issue-labeler bot added the area-crossgen2-coreclr label Aug 26, 2024

dotnet-policy-service bot added the community-contribution Indicates that the PR has been added by a community member label Aug 26, 2024

jakobbotsch reviewed Aug 26, 2024

View reviewed changes

src/tests/Common/Directory.Build.targets Outdated Show resolved Hide resolved

am11 force-pushed the feature/tests/published-crossgen2 branch from 4c7e915 to aec8452 Compare August 27, 2024 09:57

am11 mentioned this pull request Aug 27, 2024

Test failure: baseservices/TieredCompilation/BasicTestWithMcj/BasicTestWithMcj.cmd #106949

Closed

Revert Revert "Copy published crossgen2 in artifacts/tests (dotnet#80154

788302f

)" This reverts commit b579962.

am11 force-pushed the feature/tests/published-crossgen2 branch 2 times, most recently from b67985f to 2d90316 Compare August 27, 2024 18:51

Skip crossgen2 publishing for comparison jobs etc.

8fff36a

am11 force-pushed the feature/tests/published-crossgen2 branch from 2d90316 to 8fff36a Compare August 27, 2024 18:52

Pass LibrariesConfiguration to <MSBuild>

e055a59

Rename to UsePublishedCrossgen2

9a1016f

Merge branch 'main' into feature/tests/published-crossgen2

cc8f619

am11 marked this pull request as ready for review August 28, 2024 12:22

Update superpmi.py

334e759

Merge branch 'main' into feature/tests/published-crossgen2

d2e195a

jakobbotsch approved these changes Sep 2, 2024

View reviewed changes

jakobbotsch merged commit b234aef into dotnet:main Sep 2, 2024
151 of 155 checks passed

am11 deleted the feature/tests/published-crossgen2 branch September 2, 2024 11:44

build-analysis bot mentioned this pull request Sep 2, 2024

System.Numerics.Tensors.Tests.TensorSpanTests.TensorExtensionsTwoSpanInFloatOut fails with IndexOutOfRangeException #107254

Open

clamp03 mentioned this pull request Sep 6, 2024

Fix published crossgen2 crash #107447

Closed

a74nh mentioned this pull request Sep 6, 2024

Cannot test on Arm64 : Could not copy crossgen2_publish apphost #107457

Closed

BruceForstall mentioned this pull request Sep 10, 2024

Building CORE_ROOT is broken #107632

Closed

jkotas mentioned this pull request Sep 14, 2024

Use live apphost when publishing ilc as singlefile #105004

Open

jtschuster pushed a commit to jtschuster/runtime that referenced this pull request Sep 17, 2024

Revert Revert "Copy published crossgen2 in artifacts/tests (dotnet#80154

091c664

)" (dotnet#106965) This reverts commit b579962.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Revert Revert "Copy published crossgen2 in artifacts/tests (#80154)" #106965

Revert Revert "Copy published crossgen2 in artifacts/tests (#80154)" #106965

am11 commented Aug 26, 2024 •

edited

Loading

am11 commented Aug 26, 2024

jakobbotsch commented Aug 26, 2024

jakobbotsch commented Aug 26, 2024

azure-pipelines bot commented Aug 26, 2024

jakobbotsch commented Aug 26, 2024

azure-pipelines bot commented Aug 26, 2024

jakobbotsch commented Aug 26, 2024

azure-pipelines bot commented Aug 26, 2024

jakobbotsch commented Aug 27, 2024

am11 commented Aug 27, 2024

am11 commented Aug 27, 2024 •

edited

Loading

am11 commented Aug 27, 2024

am11 commented Aug 27, 2024

jakobbotsch commented Aug 27, 2024

azure-pipelines bot commented Aug 27, 2024

am11 commented Aug 28, 2024

am11 commented Aug 28, 2024

jkotas commented Aug 28, 2024

jakobbotsch commented Aug 28, 2024

am11 commented Aug 28, 2024

jkotas commented Aug 29, 2024

jakobbotsch commented Aug 29, 2024

azure-pipelines bot commented Aug 29, 2024

jakobbotsch commented Aug 29, 2024

am11 commented Aug 29, 2024

jakobbotsch commented Aug 30, 2024

am11 commented Aug 30, 2024

jakobbotsch commented Aug 30, 2024

azure-pipelines bot commented Aug 30, 2024

am11 commented Aug 31, 2024

jakobbotsch commented Sep 2, 2024

Revert Revert "Copy published crossgen2 in artifacts/tests (#80154)" #106965

Revert Revert "Copy published crossgen2 in artifacts/tests (#80154)" #106965

Conversation

am11 commented Aug 26, 2024 • edited Loading

am11 commented Aug 26, 2024

jakobbotsch commented Aug 26, 2024

jakobbotsch commented Aug 26, 2024

azure-pipelines bot commented Aug 26, 2024

jakobbotsch commented Aug 26, 2024

azure-pipelines bot commented Aug 26, 2024

jakobbotsch commented Aug 26, 2024

azure-pipelines bot commented Aug 26, 2024

jakobbotsch commented Aug 27, 2024

am11 commented Aug 27, 2024

am11 commented Aug 27, 2024 • edited Loading

am11 commented Aug 27, 2024

am11 commented Aug 27, 2024

jakobbotsch commented Aug 27, 2024

azure-pipelines bot commented Aug 27, 2024

am11 commented Aug 28, 2024

am11 commented Aug 28, 2024

jkotas commented Aug 28, 2024

jakobbotsch commented Aug 28, 2024

am11 commented Aug 28, 2024

jkotas commented Aug 29, 2024

jakobbotsch commented Aug 29, 2024

azure-pipelines bot commented Aug 29, 2024

jakobbotsch commented Aug 29, 2024

am11 commented Aug 29, 2024

jakobbotsch commented Aug 30, 2024

am11 commented Aug 30, 2024

jakobbotsch commented Aug 30, 2024

azure-pipelines bot commented Aug 30, 2024

am11 commented Aug 31, 2024

jakobbotsch commented Sep 2, 2024

am11 commented Aug 26, 2024 •

edited

Loading

am11 commented Aug 27, 2024 •

edited

Loading