tools.sh fails with core dump while building runtime

[ellahathaway]

This build is failing with the following error while building runtime:

Unhandled exception. Microsoft.Build.Framework.InternalErrorException: MSB0001: Internal MSBuild Error: must be valid
       at Microsoft.Build.Shared.ErrorUtilities.ThrowInternalError(String message, Exception innerException, Object[] args)
       at Microsoft.Build.BackEnd.Logging.LoggingContext.LogBuildEvent(BuildEventArgs buildEvent)
       at Microsoft.Build.BackEnd.Components.RequestBuilder.AssemblyLoadsTracker.CurrentDomainOnAssemblyLoad(Object sender, AssemblyLoadEventArgs args)
    /vmr/src/runtime/eng/common/tools.sh: line 499: 64596 Aborted                 (core dumped) "$_InitializeBuildTool" "$@"
    Build failed with exit code 134. Check errors above.

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[ellahathaway]

This did not occur in the latest build. It's possible that it is either 1) fixed or 2) an inconsistent error. Nothing in the diff between the failing build and the successful build indicates that a fix went in: dotnet/dotnet@03b2837...5b7c2cf

[mthalman]

This is similar to #4515. @JanKrivanek - Has there been any progress in identifying the cause? There have been a lot of these MSBuild errors that have been popping up randomly from LoggingContext.

[ellahathaway]

Looks like dotnet/msbuild#10342 describes the same msbuild error. This issue was closed with dotnet/msbuild#10401, so we may just have to wait for this change to flow into the VMR.

[MichaelSimons]

FYI, The msbuild change flowed in with dotnet/dotnet@445936c.

[ellahathaway]

The error appeared again in a build from yesterday at 3:46 AM (internal Microsoft link), but this time the error occurred while building vstest. The build of the VMR contains the msbuild commit hash from dotnet/dotnet@445936c, so it seems that dotnet/msbuild#10401 did not solve the issue.

MSBUILD : error MSB4166: UNHANDLED EXCEPTIONS FROM PROCESS 86800:
    MSBUILD : error MSB4166: =====================
    MSBUILD : error MSB4166: 07/21/2024 11:52:15
    MSBUILD : error MSB4166: Microsoft.Build.Framework.InternalErrorException: MSB0001: Internal MSBuild Error: must be valid
    MSBUILD : error MSB4166:    at Microsoft.Build.Shared.ErrorUtilities.ThrowInternalError(String message, Exception innerException, Object[] args)
    MSBUILD : error MSB4166:    at Microsoft.Build.BackEnd.Logging.LoggingContext.LogBuildEvent(BuildEventArgs buildEvent)
    MSBUILD : error MSB4166:    at Microsoft.Build.BackEnd.Components.RequestBuilder.AssemblyLoadsTracker.CurrentDomainOnAssemblyLoad(Object sender, AssemblyLoadEventArgs args)
    MSBUILD : error MSB4166: ===================
    MSBUILD : error MSB4166: 
    MSBUILD : error MSB4166: 

cc @JanKrivanek

[MichaelSimons]

I haven't been following this situation closely... Does this situation require a rebootstrap in order to pick up the fix in the toolset we build with?

[JanKrivanek]

The error message must have been produced by the older version of MSBuild (the message wording was changed on Jul/11th: dotnet/msbuild#10348, the stack trace shown was eliminated Jul/18th: dotnet/msbuild#10401).

Can we doublecheck the MSBuild version used?

[ellahathaway]

The binlog shows that the MSBuild version is 17.12.0-preview-24354-01+8c76ec59b and the MSBuild executable path as /vmr/.dotnet/sdk/9.0.100-preview.7.24357.1/MSBuild.dll. Based on the fact that it's using the msbuild assembly from the sdk, we need to rebootstrap to successfully resolve the failures we're experiencing.

[ellahathaway]

Update on the rebootstrapping:

The pipeline run that I needed to do a rebootstrap succeeded, but our rebootstrap pipeline is down due to service connection issues. I no longer have permissions to upload artifacts manually, so I can't rebootstrap without our pipeline.

I've pinged FR for help in the meantime.

[ellahathaway]

I haven't seen this issue resurface since we rebootstrapped. Because of that, I'm closing this issue.