"The process cannot access the file" error sporadically but frequently in CI builds

[stephentoub]

I've been seeing CI legs failing sporadically but frequently with an error like this during the build:

  The process cannot access the file 'F:\workspace\_work\1\s\artifacts\SymStore\Debug\System.Security.Cryptography.Pkcs\netcoreapp5.0\System.Security.Cryptography.Pkcs.pdb' because it is being used by another process.
F:\workspace\_work\1\s\.packages\microsoft.dotnet.arcade.sdk\5.0.0-beta.20112.7\tools\SymStore.targets(61,5): error MSB3073: The command ""F:\workspace\_work\1\s\.packages\microsoft.diasymreader.pdb2pdb\1.1.0-beta2-19575-01\tools\Pdb2Pdb.exe" "F:\workspace\_work\1\s\artifacts\bin\System.Security.Cryptography.Pkcs\netcoreapp5.0-Windows_NT-Debug\System.Security.Cryptography.Pkcs.dll" /out "F:\workspace\_work\1\s\artifacts\SymStore\Debug\System.Security.Cryptography.Pkcs\netcoreapp5.0\System.Security.Cryptography.Pkcs.pdb" /srcsvrvar SRC_INDEX=public" exited with code 2. [F:\workspace\_work\1\s\src\libraries\System.Security.Cryptography.Pkcs\src\System.Security.Cryptography.Pkcs.csproj]
##[error].packages\microsoft.dotnet.arcade.sdk\5.0.0-beta.20112.7\tools\SymStore.targets(61,5): error MSB3073: The command ""F:\workspace\_work\1\s\.packages\microsoft.diasymreader.pdb2pdb\1.1.0-beta2-19575-01\tools\Pdb2Pdb.exe" "F:\workspace\_work\1\s\artifacts\bin\System.Security.Cryptography.Pkcs\netcoreapp5.0-Windows_NT-Debug\System.Security.Cryptography.Pkcs.dll" /out "F:\workspace\_work\1\s\artifacts\SymStore\Debug\System.Security.Cryptography.Pkcs\netcoreapp5.0\System.Security.Cryptography.Pkcs.pdb" /srcsvrvar SRC_INDEX=public" exited with code 2.

e.g. https://dev.azure.com/dnceng/public/_build/results?buildId=520739&view=logs&jobId=423ae147-26a3-5a4d-a4d3-4b3fc1dd598e&j=423ae147-26a3-5a4d-a4d3-4b3fc1dd598e&t=3d4a7aa4-0a94-53e5-e198-62555696ed38
cc: @ViktorHofer

[ViktorHofer]

@tmat possibly a race in SymStore.targets?

[tmat]

The error is caused by converted PDBs for two different target frameworks being generated into the same directory.
SymStore.targets uses $(TargetFramework) in the output directory path. The value of TargetFramework when the target runs is netcoreapp5.0, not netcoreapp5.0-Windows_NT.

It seems that the property value is split on - and reassigned here:
microsoft.dotnet.build.tasks.targetframework.sdk\5.0.0-beta.20112.7\build\Microsoft.DotNet.Build.Tasks.TargetFramework.Sdk.props

[ViktorHofer]

Thanks @tmat, you are right, our TargetFramework properties aren't unique per build. @ercistj @Anipik, suggestions?

[Anipik]

we should fix it by overriding the symbols path to include OSGroup similar to what we do for output and intermediate paths.

[ViktorHofer]

In this case sure, even though we should probably avoid using OSGroup which is a custom property in dotnet/runtime, I would prefer TargetFrameworkSuffix. How would we fix other instances where we don't own the sources and can't hook into them? Can we avoid such breaks by coming up with a better pattern which guarantees unique TargetFramework properties in the different inner-builds?

[Anipik]

Yes i should have been more clear, we use TargetFrameworkSuffix for paths. We should use that only.

Can we avoid such breaks by coming up with a better pattern which guarantees unique TargetFramework properties in the different inner-builds?

Actually, That was our first experiment. Later on this project we shifted to remove the TargetFrameworkSuffix from TargetFramework bcoz we hit a significant block. I will try to dig up the reason. @ericstj do u remember wht caused us to remove the TargetFrameworkSuffix during the build ?

How would we fix other instances where we don't own the sources and can't hook into them?

we could move the output path calculation stuff to sdk rather in the runtime repo and then anybody using the sdk wouldnt require to update the paths.

[ViktorHofer]

we could move the output path calculation stuff to sdk rather in the runtime repo and then anybody using the sdk wouldnt require to update the paths.

That wouldn't have helped in this situation as the path is calculated in an external component without using pre-defined variables like (Base)OutputPath or (Base)IntermediateOutputPath: https://github.com/dotnet/arcade/blob/2086e534f12e6116889ee480646ef54c1f781887/src/Microsoft.DotNet.Arcade.Sdk/tools/SymStore.targets#L28

Actually, That was our first experiment.

+1. Yes, please dig up the reason for the change in direction.

[tmat]

Also, how does this gonna work with the new .NET SDK feature that also adds platform specification to the TFM?

[ViktorHofer]

Happened again: https://dev.azure.com/dnceng/public/_build/results?buildId=524971&view=logs

[ViktorHofer]

Also, how does this gonna work with the new .NET SDK feature that also adds platform specification to the TFM?

That's still tbd but using a different separator should be fine as well. In the meantime I propose to fix this by (conditionally) adding the TargetFrameworkSuffix property to SymStore.targets.

[tmat]

@ViktorHofer Sounds good, as long as that's just a temporary workaround. As you pointed out Microsoft.DotNet.Build.Tasks.TargetFramework.Sdk.props breaks invariant that holds in standard SDK build.