build.cmd fails with latest VS 2022 dogfood

[vitek-karas]

I updated my VS 2022 preview to 17.0.0 Preview 5.0 [31805.14.d17.0] and running build.cmd fails now with:

  Send-VsDevShellTelemetry : Could not load file or assembly 'Newtonsoft.Json, Version=13.0.0.0, Culture=neutral,
  PublicKeyToken=30ad4fe6b2a6aeed' or one of its dependencies. The system cannot find the file specified.
  At line:1 char:168
  + ... udio.DevShell.dll'; Send-VsDevShellTelemetry -NewInstanceType Cmd; }}
  +                         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      + CategoryInfo          : NotSpecified: (:) [Send-VsDevShellTelemetry], FileNotFoundException
      + FullyQualifiedErrorId : System.IO.FileNotFoundException,Microsoft.VisualStudio.DevShell.Commands.SendVsDevShellT

     elemetryCommand

..\runtime\src\coreclr\ToolBox\SOS\DacTableGen\DacTableGen.csproj(22,5): error : VSINSTALLDIR environment variable was not set correctly. You may need to repair Visual Studio installation.

Running the affected eng\native\init-vs-end.vmd from clean command line works just fine, so this has to be something with the environment inside the build.

I was able to workaround the problem by setting VSCMD_SKIP_SENDTELEMETRY=1 which skips over the problematic piece of code in VsDevCmd.bat.

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[ghost]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

---

I updated my VS 2022 preview to 17.0.0 Preview 5.0 [31805.14.d17.0] and running build.cmd fails now with:

  Send-VsDevShellTelemetry : Could not load file or assembly 'Newtonsoft.Json, Version=13.0.0.0, Culture=neutral,
  PublicKeyToken=30ad4fe6b2a6aeed' or one of its dependencies. The system cannot find the file specified.
  At line:1 char:168
  + ... udio.DevShell.dll'; Send-VsDevShellTelemetry -NewInstanceType Cmd; }}
  +                         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      + CategoryInfo          : NotSpecified: (:) [Send-VsDevShellTelemetry], FileNotFoundException
      + FullyQualifiedErrorId : System.IO.FileNotFoundException,Microsoft.VisualStudio.DevShell.Commands.SendVsDevShellT

     elemetryCommand

..\runtime\src\coreclr\ToolBox\SOS\DacTableGen\DacTableGen.csproj(22,5): error : VSINSTALLDIR environment variable was not set correctly. You may need to repair Visual Studio installation.

Running the affected eng\native\init-vs-end.vmd from clean command line works just fine, so this has to be something with the environment inside the build.

I was able to workaround the problem by setting VSCMD_SKIP_SENDTELEMETRY=1 which skips over the problematic piece of code in VsDevCmd.bat.

Author:	vitek-karas
Assignees:	-
Labels:	area-Infrastructure, untriaged
Milestone:	-

[janvorli]

I have just hit the same problem.

[jkotas]

I have opened https://dev.azure.com/devdiv/DevDiv/_workitems/edit/1416950

[danmoseley]

Email the VS dog food SWAT alias perhaps? They move quickly if issue is blocking dog food.

[jkotas]

Yes, they are on it already.

[trippwill]

We definitely have an error in VsDevCmd.bat which I'm investigating related to loading of dependent assemblies.

However the telemetry sending portion is designed to be fault-tolerant. When run normally, the command to send telemetry if the user is opted-in is started in a seperate process without a window and all output redirected from the START command is dumped to NUL. Then VsDevCmd.bat continues and terminates as usual. The user should never see an error if one occurs.

This changes if the env var VSCMD_DEBUG is set to any value in the parent environment. In this case, the telemetry command is executed in the same scope and process, so that the output can be seen when debugging VsDevCmd.bat. @vitek-karas, was VSCMD_DEBUG set when you called build.cmd?

[jkotas]

VSCMD_DEBUG is not set when this is failing in this repo build environment.

VsDevCmd.bat is called via nested chain of cmd, msbuild and powershell invocations. This nesting makes the error leak to stderr somehow. The build scripting treats writes to stderr as build errors, and fails the build.

[jkotas]

@trippwill The problem seems to be that START "" /B powershell.exe ... done by VsDevCmd.bat is not guaranteed to send its output to /dev/null. It sometimes sends its output to the default stdout/stderr of the VsDevCmd.bat script.

[trippwill]

@jkotas thanks for the info. I just did a bit more research and it looks like the inner command of START is still hooked up to stdin/stderr, and has to be redirected seperately. In our "usual" use case, this isn't a big deal, but if stderr is being monitored then errors still go there.

We should be able to fix the underlying error though. And then I can look at the issue of redirecting stderr as a distinct issue.

[jkotas]

This is fixed in latest VS 2022 dogfood.

[eiriktsarpalis]

I've started seeing this issue in the dogfood builds of the last few days (reproducing with Version 17.1.0 Preview 2.0 [31831.6.main]). The workaround suggested in the OP resolves the issue for me.

[vitek-karas]

Same here on Version 17.1.0 Preview 2.0 [31831.6.main] as well.
I reactivated the VS issue: https://dev.azure.com/devdiv/DevDiv/_workitems/edit/1416950

[vitek-karas]

I also emailed the Dogfood SWAT alias.