Multi-proc build of a .sln file is 'The process cannot access the file because it is being used by another process.' #739
Comments
|
@srivatsn @dsplaisted @nguerrera This seems like an issue we'd want to fix for "1.0". |
|
Looks like a reoccurrence of #475 . I'll see if I can get a repro |
|
We see this in our CLI Jenkins runs. Here's another: @natidea - let me know if you need help repro'ing it. On my machine it happens pretty consistently. |
|
CLI Jenkins CI is now hitting this issues multiple times on each PR. Likely running |
|
FYI, we're trying to disable multi-proc builds of the affected SLN in CLI until the base issue gets resolved. Once that merge goes through, you will want to delete this line when attempting to repro: |
|
I have a repro that I am looking at this weekend. Also /cc @rainersigwald @nguerrera |
|
Do we have any insight on what source changes were made when this started happening? Was it an upgrade of stage0 sdk? From what build to what build? |
|
@piotrpMSFT just changed our build script this week to build the test sln file instead of globbing *.csproj under the test directory and building each project separately. The error doesn't happen consistently, so it made it through our CI, but then started cropping up. |
|
I didn't see how the globs were processed before in that change. Were they batched serially or passed to msbuild all at once? Trying to understand if true parallelism was really only first introduced with that change or there really is a difference in sln build case |
|
Here's how the test projects were found: <PreTestProjects Include="test$(PathSeparator)*$(PathSeparator)*.csproj;" />
<TestProjects Include="%(PreTestProjects.Fullpath)"
Exclude="@(TestProjectsToExclude)" />And then here's how the tests were built: https://github.com/dotnet/cli/pull/5403/files#diff-70eb062c7648e65aa7aff25e924d014bL77 <Target Name="BuildTests"
DependsOnTargets="RestoreTests;
SetupTestProjectData;"
Inputs="%(TestProjects.BuildInputs)"
Outputs="%(TestProjects.BuildOutput)">
<DotNetBuild Configuration="$(Configuration)"
Framework="%(TestProjects.Framework)"
ToolPath="$(Stage0Directory)"
ProjectPath="%(TestProjects.ProjectPath)" />
</Target> |
|
Ok they were serialized before |
|
Yes, they were serialized. We originally tried parallel build but hit issues due to shared dependencies. Tried sln build as a dogfooding and code cleanup exercise. Reducing build nodes to 1 seems to have mitigated but it should not be required... |
|
I have a The interesting things to me are these two entries: and and then that last task is failing: As you can see, node 16 and node 22 are trying to compile the same project to the same output file roughly 35ms apart. @rainersigwald @jeffkl @cdmihai - doesn't MSBuild have a locking mechanism so 2 nodes don't try building the same project at the same time? |
|
No, the rule is that the engine builds only once for each tuple of:
It's definitely concerning that something in the build is building project 4 22 times. I'm going to apply some of my magic internal tracing tools to the Sdk repo to help chase this down. |
|
I am running into this reliably if I target the my .sln file. Targeting the .csproj build correctly. I can provide logs if it will help. |
|
@AlgorithmsAreCool in the CLI repo or another public repo? Unfortunately, standard logging is hard to decipher for this problem. I have a still-locked-inside-the-firewall tool that can help, but to use it I need to be able to build using full framework MSBuild. If you have a public repo that hits this I could take a look at it. |
|
I'm afraid the code is private, I can try to put together a minimal repro and upload it on here. I should be able to get to that early tomorrow. |
|
@AlgorithmsAreCool I think the relevant information should just be: what projects do you have, what is the project-to-project reference structure, and what are the project's TFMs? |
|
Ok, my "omg 22" business above may have been a bit premature. I'm throwing together a logger to help with debugging this sort of issue: https://github.com/rainersigwald/ParallelBuildDebuggingLogger It turns out that many/most of the requests will be because of the determine-tfm-from-requesting-tfm dance. For example, on a .sln with 2 projects each dual targeted, I see: There are many flavors of invocation of both projects, but CoreCompile is (correctly) invoked only twice per project. Now I'll try this on the CLI repo where this problem is being observed and see if anything stands out. |
|
@rainersigwald I'm still trying to get a minimal repro working, I can tickle the issue but not hit it every time. My project structure is :
Also the error always is or a warning of: |
|
That's a great repro, @AlgorithmsAreCool, thanks! This shows that the race is between
That explains a few questions I had ("why only in solutions?" and "why can't I repro this with my multitargeting attempts?"). @nguerrera @natidea I think we can fix this without going all the way to fixing dotnet/msbuild#1276 (which would probably be ideal but is invasive). Would it be reasonable to respond to the |
|
We stopped returning "no information" for single targeting because there was a bug where if caller had TargetFramework set as global property (say caller is multi-targeted), then it would be used instead of TargetFramework of callee. Is there something we can do with TreatAsLocal that could help with this instead? |
|
Does |
|
We only remove it from the call to GetTargetFrameworkProperties, but not subsequent targets. Does it work to remove a property and pass it? If so, we could remove it from all calls. |
|
Well this is clear as mud. Looks like So that won't work. Could we instead unset the property by passing |
|
Will that allow the project to reset it to non empty? |
|
The Current notion:
Some serious hoop-jumping here, though. Starting to wonder if it'd be better to go with the proposed solution to dotnet/msbuild#1276 that would allow all existing project and all new single-tfm projects to just return "build me directly", and only invoke inner builds on new multi-tfm projects (which would match the inner builds driven by the call from the sln to the outer build). That is also a scary change, though. |
|
Another notion: from a single-tfm project, return both the tfm and a flag that tells the consuming project to set the metadata |
|
I have proof of concept on that latter option. diff --git "a/c:\\Program Files\\dotnet - Copy\\sdk\\1.0.0-rc4-004689/Microsoft.Common.CurrentVersion.targets" "b/C:\\Program Files\\dotnet\\sdk\\1.0.0-rc4-004689/Microsoft.Common.CurrentVersion.targets"
index da772b0..7f82f2a 100644
--- "a/c:\\Program Files\\dotnet - Copy\\sdk\\1.0.0-rc4-004689/Microsoft.Common.CurrentVersion.targets"
+++ "b/C:\\Program Files\\dotnet\\sdk\\1.0.0-rc4-004689/Microsoft.Common.CurrentVersion.targets"
@@ -1527,9 +1527,18 @@ Copyright (C) Microsoft Corporation. All rights reserved.
<Output TaskParameter="TargetOutputs" PropertyName="_ProjectReferenceTargetFrameworkProperties" />
</MSBuild>
+ <PropertyGroup>
+ <_RemoveTargetFrameworkFromReference Condition="'$(_ProjectReferenceTargetFrameworkProperties.Contains(`ProjectHasSingleTargetFramework=true`))'">true</_RemoveTargetFrameworkFromReference>
+ </PropertyGroup>
+
<ItemGroup>
<_MSBuildProjectReferenceExistent Condition="'%(_MSBuildProjectReferenceExistent.Identity)' == '%(Identity)'">
<SetTargetFramework>$(_ProjectReferenceTargetFrameworkProperties)</SetTargetFramework>
+
+ <!-- Unconditionally remove the property that was set as a marker to indicate that for this call we should remove TargetFramework -->
+ <UndefineProperties>%(_MSBuildProjectReferenceExistent.UndefineProperties);ProjectHasSingleTargetFramework</UndefineProperties>
+
+ <UndefineProperties Condition="'$(_RemoveTargetFrameworkFromReference)' == 'true'">%(_MSBuildProjectReferenceExistent.UndefineProperties);TargetFramework</UndefineProperties>
</_MSBuildProjectReferenceExistent>
</ItemGroup>
diff --git "a/c:\\Program Files\\dotnet - Copy\\sdk\\1.0.0-rc4-004689/Sdks/Microsoft.NET.Sdk/build/Microsoft.NET.Sdk.Common.targets" "b/C:\\Program Files\\dotnet\\sdk\\1.0.0-rc4-004689/Sdks/Microsoft.NET.Sdk/build/Microsoft.NET.Sdk.Common.targets"
index c1c71c2..de62c6c 100644
--- "a/c:\\Program Files\\dotnet - Copy\\sdk\\1.0.0-rc4-004689/Sdks/Microsoft.NET.Sdk/build/Microsoft.NET.Sdk.Common.targets"
+++ "b/C:\\Program Files\\dotnet\\sdk\\1.0.0-rc4-004689/Sdks/Microsoft.NET.Sdk/build/Microsoft.NET.Sdk.Common.targets"
@@ -55,13 +55,16 @@ Copyright (c) .NET Foundation. All rights reserved.
with the referencing project's target framework.
============================================================
-->
- <Target Name="GetTargetFrameworkProperties" Returns="TargetFramework=$(NearestTargetFramework)">
+ <Target Name="GetTargetFrameworkProperties" Returns="TargetFramework=$(NearestTargetFramework);ProjectHasSingleTargetFramework=$(_HasSingleTargetFramework)">
<PropertyGroup>
<!-- If a ReferringTargetFramework was not specified, and we only have one TargetFramework, then don't try to check compatibility -->
<_SkipNearestTargetFrameworkResolution Condition="'$(TargetFramework)' != '' and '$(ReferringTargetFramework)' == ''">true</_SkipNearestTargetFrameworkResolution>
<NearestTargetFramework Condition="'$(_SkipNearestTargetFrameworkResolution)' == 'true'">$(TargetFramework)</NearestTargetFramework>
+ <_HasSingleTargetFramework Condition="'$(TargetFramework)' != '' and '$(TargetFrameworks)' == ''">true</_HasSingleTargetFramework>
+ <_HasSingleTargetFramework Condition="'$(_HasSingleTargetFramework)' == ''">false</_HasSingleTargetFramework>
+
<_PossibleTargetFrameworks Condition="'$(TargetFramework)' != ''">$(TargetFramework)</_PossibleTargetFrameworks>
<_PossibleTargetFrameworks Condition="'$(TargetFramework)' == ''">$(TargetFrameworks)</_PossibleTargetFrameworks>
</PropertyGroup>@nguerrera thoughts? |
|
simplified: diff --git "a/c:\\Program Files\\dotnet - Copy\\sdk\\1.0.0-rc4-004689/Microsoft.Common.CurrentVersion.targets" "b/C:\\Program Files\\dotnet\\sdk\\1.0.0-rc4-004689/Microsoft.Common.CurrentVersion.targets"
index da772b0..4a60364 100644
--- "a/c:\\Program Files\\dotnet - Copy\\sdk\\1.0.0-rc4-004689/Microsoft.Common.CurrentVersion.targets"
+++ "b/C:\\Program Files\\dotnet\\sdk\\1.0.0-rc4-004689/Microsoft.Common.CurrentVersion.targets"
@@ -1530,6 +1530,10 @@ Copyright (C) Microsoft Corporation. All rights reserved.
<ItemGroup>
<_MSBuildProjectReferenceExistent Condition="'%(_MSBuildProjectReferenceExistent.Identity)' == '%(Identity)'">
<SetTargetFramework>$(_ProjectReferenceTargetFrameworkProperties)</SetTargetFramework>
+
+ <UndefineProperties Condition="$(_ProjectReferenceTargetFrameworkProperties.Contains(`ProjectHasSingleTargetFramework=true`))">%(_MSBuildProjectReferenceExistent.UndefineProperties);TargetFramework;ProjectHasSingleTargetFramework</UndefineProperties>
+ <!-- Unconditionally remove the property that was set as a marker to indicate that for this call we should remove TargetFramework -->
+ <UndefineProperties Condition="!$(_ProjectReferenceTargetFrameworkProperties.Contains(`ProjectHasSingleTargetFramework=true`))">%(_MSBuildProjectReferenceExistent.UndefineProperties);ProjectHasSingleTargetFramework</UndefineProperties>
</_MSBuildProjectReferenceExistent>
</ItemGroup>
diff --git "a/c:\\Program Files\\dotnet - Copy\\sdk\\1.0.0-rc4-004689/Sdks/Microsoft.NET.Sdk/build/Microsoft.NET.Sdk.Common.targets" "b/C:\\Program Files\\dotnet\\sdk\\1.0.0-rc4-004689/Sdks/Microsoft.NET.Sdk/build/Microsoft.NET.Sdk.Common.targets"
index c1c71c2..de62c6c 100644
--- "a/c:\\Program Files\\dotnet - Copy\\sdk\\1.0.0-rc4-004689/Sdks/Microsoft.NET.Sdk/build/Microsoft.NET.Sdk.Common.targets"
+++ "b/C:\\Program Files\\dotnet\\sdk\\1.0.0-rc4-004689/Sdks/Microsoft.NET.Sdk/build/Microsoft.NET.Sdk.Common.targets"
@@ -55,13 +55,16 @@ Copyright (c) .NET Foundation. All rights reserved.
with the referencing project's target framework.
============================================================
-->
- <Target Name="GetTargetFrameworkProperties" Returns="TargetFramework=$(NearestTargetFramework)">
+ <Target Name="GetTargetFrameworkProperties" Returns="TargetFramework=$(NearestTargetFramework);ProjectHasSingleTargetFramework=$(_HasSingleTargetFramework)">
<PropertyGroup>
<!-- If a ReferringTargetFramework was not specified, and we only have one TargetFramework, then don't try to check compatibility -->
<_SkipNearestTargetFrameworkResolution Condition="'$(TargetFramework)' != '' and '$(ReferringTargetFramework)' == ''">true</_SkipNearestTargetFrameworkResolution>
<NearestTargetFramework Condition="'$(_SkipNearestTargetFrameworkResolution)' == 'true'">$(TargetFramework)</NearestTargetFramework>
+ <_HasSingleTargetFramework Condition="'$(TargetFramework)' != '' and '$(TargetFrameworks)' == ''">true</_HasSingleTargetFramework>
+ <_HasSingleTargetFramework Condition="'$(_HasSingleTargetFramework)' == ''">false</_HasSingleTargetFramework>
+
<_PossibleTargetFrameworks Condition="'$(TargetFramework)' != ''">$(TargetFramework)</_PossibleTargetFrameworks>
<_PossibleTargetFrameworks Condition="'$(TargetFramework)' == ''">$(TargetFrameworks)</_PossibleTargetFrameworks>
</PropertyGroup> |
|
That looks promising. |
rainersigwald
added a commit
to rainersigwald/msbuild
that referenced
this issue
Feb 2, 2017
Partially fixes dotnet/sdk#739. The first iteration of cross-targeting support code unconditionally queried each ProjectReference for the best TFM to build against, and then explicitly specified that TFM when building the reference. This caused a race condition when building a set of projects that had a single-TFM project and another project that had a reference to it. The entry point (probably .sln) build would build the referenced project with no TF specified, and then the referencing project would build it with an explicit TF specified. These two builds appeared different to the MSBuild engine (because they had different sets of global properties) and so were both executed, resulting in a race condition. The fix is in two parts: * Allow a project to report back through GetTargetFrameworkProperties that it only has one TF to build. * When a project reports that it has only one TF to build, issue its build request without specifying any TF. This commit is the latter.
rainersigwald
added a commit
to rainersigwald/sdk
that referenced
this issue
Feb 2, 2017
Partially fixes dotnet#739. The first iteration of cross-targeting support code unconditionally queried each ProjectReference for the best TFM to build against, and then explicitly specified that TFM when building the reference. This caused a race condition when building a set of projects that had a single-TFM project and another project that had a reference to it. The entry point (probably .sln) build would build the referenced project with no TF specified, and then the referencing project would build it with an explicit TF specified. These two builds appeared different to the MSBuild engine (because they had different sets of global properties) and so were both executed, resulting in a race condition. The fix is in two parts: * Allow a project to report back through GetTargetFrameworkProperties that it only has one TF to build. * When a project reports that it has only one TF to build, issue its build request without specifying any TF. This commit is the former.
This was referenced Feb 2, 2017
AndyGerlicher
pushed a commit
to dotnet/msbuild
that referenced
this issue
Feb 3, 2017
Partially fixes dotnet/sdk#739. The first iteration of cross-targeting support code unconditionally queried each ProjectReference for the best TFM to build against, and then explicitly specified that TFM when building the reference. This caused a race condition when building a set of projects that had a single-TFM project and another project that had a reference to it. The entry point (probably .sln) build would build the referenced project with no TF specified, and then the referencing project would build it with an explicit TF specified. These two builds appeared different to the MSBuild engine (because they had different sets of global properties) and so were both executed, resulting in a race condition. The fix is in two parts: * Allow a project to report back through GetTargetFrameworkProperties that it only has one TF to build. * When a project reports that it has only one TF to build, issue its build request without specifying any TF. This commit is the latter.
|
If need additional code for repro/verification, this bug happens consistently on https://github.com/aspnet/DependencyInjection. All you have to do is Using 1.0.0-rc4-004757 |
rainersigwald
added a commit
to rainersigwald/sdk
that referenced
this issue
Feb 3, 2017
Partially fixes dotnet#739. The first iteration of cross-targeting support code unconditionally queried each ProjectReference for the best TFM to build against, and then explicitly specified that TFM when building the reference. This caused a race condition when building a set of projects that had a single-TFM project and another project that had a reference to it. The entry point (probably .sln) build would build the referenced project with no TF specified, and then the referencing project would build it with an explicit TF specified. These two builds appeared different to the MSBuild engine (because they had different sets of global properties) and so were both executed, resulting in a race condition. The fix is in two parts: * Allow a project to report back through GetTargetFrameworkProperties that it only has one TF to build. * When a project reports that it has only one TF to build, issue its build request without specifying any TF (dotnet/msbuild#1667) This commit is the former.
|
This bug is marked as fixed in 1.0 RTM yet we are seeing it on Rosyln using 1.01 builds.
The problem reproduces locally on some machines, but not others and only on Mac. Our Linux runs which use the same testing scripts don't hit this issue. CC @khyperia |
|
We are hitting this multiple times a day in roslyn. Why is this issue still marked as closed? Is anyone looking into it??? |
|
It is closed because #797 paired with dotnet/msbuild#1667 fixed a specific race condition that was very common. I think you've hit a different race with same error message symptom. |
|
The fact that it is only occurring on Mac is very different from the original reports. |
|
Let's use #475 to track. |
|
It is happening on windows too https://ci.dot.net/job/dotnet_roslyn/job/master/job/windows_determinism_prtest/9548/ |
|
See #475 (comment). |
mmitche
pushed a commit
to mmitche/sdk
that referenced
this issue
Jun 5, 2020
…0190629.2 (dotnet#739) - Microsoft.AspNetCore.Mvc.Analyzers - 3.0.0-preview7.19329.2 - Microsoft.AspNetCore.Mvc.Api.Analyzers - 3.0.0-preview7.19329.2
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
In the CLI repo, we have a Test.sln file which we
dotnet build. However, we are seeing random failures that look like the following:See https://ci.dot.net/job/dotnet_cli/job/rel_1.0.0/job/debug_windows_nt_x86_prtest/810/console
To repro:
An easier repro once you have built one time is:
F:\cli>git clean test -xdfF:\cli>.dotnet_stage0\x64\dotnet.exe restore test\Microsoft.DotNet.Cli.Tests.slnF:\cli>.dotnet_stage0\x64\dotnet.exe build test\Microsoft.DotNet.Cli.Tests.slnThe text was updated successfully, but these errors were encountered: