[Flaky Test] Roslyn.VisualStudio.IntegrationTests.CSharp.CSharpBuild.BuildWithCommandLine Test Failure

[333fred]

Failed in https://ci.dot.net/job/dotnet_roslyn/job/dev16/job/windows_release_vs-integration/24/. The logs only contain that the build failed with exit code 0, due to assert ordering. Submitted #18203 to address that issue.

[333fred]

FYI @jasonmalinowski

[333fred]

Merged #18203 so we can start seeing logs of what actually failed during tests. My PR build for that failed with the following message:

Assert.Contains() Failure
Not found: Rebuild All: 1 succeeded, 0 failed, 0 skipped
In value: 
The following files were specified on the command line:

  C:\\Users\\dotnet-bot\\AppData\\Local\\Temp\\u5jtezlr.grp\\CSharpBuild.sln

These files could not be found and will not be loaded.

[jasonmalinowski]

@jinujoseph @chborl Looks like this is the bug that's tracking that the devenv /build test is currently flaky and disabled. Let's re-enable this as a part of that other bug since it would have caught this.

[333fred]

@jasonmalinowski @chborl this failed in master yesterday, https://ci.dot.net/job/dotnet_roslyn/view/Official%20Builds/job/master/job/windows_release_vs-integration/1981/testReport/junit/Roslyn.VisualStudio.IntegrationTests.CSharp/CSharpBuild/BuildWithCommandLine/

[333fred]

And another: https://ci.dot.net/job/dotnet_roslyn/view/Official%20Builds/job/dev15.7.x/job/windows_release_vs-integration/113/testReport/junit/Roslyn.VisualStudio.IntegrationTests.CSharp/CSharpBuild/BuildWithCommandLine/

[jasonmalinowski]

@jinujoseph we might need somebody to look at this one. History has shown that if we disable this test we will break it. Repeatedly.

[sharwell]

Yep, I've seen it a few times. May be stabilized as part of #25151.

[jinujoseph]

Lets see if we hit this again after 25151

[333fred]

@jinujoseph @sharwell we hit this 4 times during merges this morning:
https://ci.dot.net/job/dotnet_roslyn/job/master/job/windows_debug_vs-integration_prtest/10595/testReport/junit/Roslyn.VisualStudio.IntegrationTests.CSharp/CSharpBuild/BuildWithCommandLine/
https://ci.dot.net/job/dotnet_roslyn/job/features_compiler/job/windows_release_vs-integration_prtest/233/testReport/junit/Roslyn.VisualStudio.IntegrationTests.CSharp/CSharpBuild/BuildWithCommandLine/
https://ci.dot.net/job/dotnet_roslyn/job/master/job/windows_debug_vs-integration_prtest/10595/testReport/junit/Roslyn.VisualStudio.IntegrationTests.CSharp/CSharpBuild/BuildWithCommandLine/
https://ci.dot.net/job/dotnet_roslyn/job/features_compiler/job/windows_release_vs-integration_prtest/233/testReport/junit/Roslyn.VisualStudio.IntegrationTests.CSharp/CSharpBuild/BuildWithCommandLine/

This needs to be looked at asap, it's failing a ton. In the meantime, I'm going to disable the test again.

[333fred]

Skip PR is merged.

[heejaechang]

let me re-enable it

[Youssef1313]

Hit this in https://github.com/dotnet/roslyn/pull/54240/checks?check_run_id=2868920846

[Youssef1313]

Hit again:

https://dev.azure.com/dnceng/public/_build/results?buildId=1201286&view=ms.vss-test-web.build-test-results-tab

[333fred]

@sharwell the flakiness here has drastically picked up over the past few weeks: the test currently has a 97% pass rating and I've seen the failure 3 times today:
https://dev.azure.com/dnceng/public/_build/results?buildId=1238555&view=ms.vss-test-web.build-test-results-tab
https://dev.azure.com/dnceng/public/_build/results?buildId=1238493&view=logs&j=922f7fe3-0735-5839-4d76-f6d8a5e047bd&t=d3c89a87-2123-56c4-3b91-63fa06060730&s=d654deb9-056d-50a2-1717-90c08683d50a (attempt 1 failed).
https://dev.azure.com/dnceng/public/_build/results?buildId=1238177&view=ms.vss-test-web.build-test-results-tab

Can you please investigate.

[sharwell]

@333fred Based on the test history, the test here isn't flaky. The failure is a true failure of devenv build to build a project. It looks like the first step to resolving this is capturing a heap dump of the devenv process which failed to complete a build and attaching it as a build artifact.

[sharwell]

@333fred This is probably related to AB#1289921. A heap dump from the deadlock is attached to the most recent build of #54851.

[allisonchou]

I opened a PR to skip this test until the underlying issue is resolved, since we're hitting this pretty often in merge flow integration test runs: #55243

[allisonchou]

Internal tracking issue (thanks Sam for link): AB#1363609

[jasonmalinowski]

@sharwell @allisonchou Should this test be re-enabled again?

[allisonchou]

Re-opening this as it still seems to be flaky.
Re-enabled in #57687.