-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Performance]: GetTargetPathWithTargetPlatformMoniker
and GetTargetPath
target post-processing can take multiple minutes on large projects.
#8985
Comments
This is very surprising behavior. I'm pretty confused as to how materializing a one-item-long list with a few metadata could be a bottleneck like you're observing. I'd like to see the binlog--to share it only with Microsoft folks, please open a feedback ticket instead and link it here so we can short-circuit the routing process. I'm also interested to see a test on 17.7-preview3 when it's available; I suspect #8747 would alleviate this by chopping off the top several methods of the callstack. But I suspect there's a deeper problem somewhere. |
@rainersigwald - The Opening a feedback ticket shortly. I'm setup to build MSBuild and I can try to do a drop-in replacement of a small set of the binaries if possible. |
I also tried disabling the de-dupe on the target to see if it would speed it up, but the list ended up with more than 200k items in it before it even made it to my project with all the references. <Target
Name="GetTargetPathWithTargetPlatformMoniker"
BeforeTargets="GetTargetPath"
DependsOnTargets="$(GetTargetPathWithTargetPlatformMonikerDependsOn)"
Returns="@(TargetPathWithTargetPlatformMoniker)"
KeepDuplicateOutputs="True">
<ItemGroup>
<TargetPathWithTargetPlatformMoniker Include="$(TargetPath)">
<TargetPlatformMoniker>$(TargetPlatformMoniker)</TargetPlatformMoniker>
<TargetPlatformIdentifier>$(TargetPlatformIdentifier)</TargetPlatformIdentifier>
<TargetFrameworkIdentifier>$(TargetFrameworkIdentifier)</TargetFrameworkIdentifier>
<TargetFrameworkVersion>$(TargetFrameworkVersion.TrimStart('vV'))</TargetFrameworkVersion>
<ReferenceAssembly Condition="'$(ProduceReferenceAssembly)' == 'true'">$(TargetRefPath)</ReferenceAssembly>
<CopyUpToDateMarker>@(CopyUpToDateMarker)</CopyUpToDateMarker>
</TargetPathWithTargetPlatformMoniker>
</ItemGroup>
</Target> |
Feedback item created and binlog attached: https://developercommunity.visualstudio.com/t/Performance:-GetTargetPathWithTargetPl/10405139 |
What list exactly? There should only be one item per project returned by GetTargetPathWithTargetPlatformMoniker/GetTargetPath, the one that is Hmm, actually could you share a memory dump in the feedback issue, maybe? |
Ah, I'm seeing something interesting in the log now. Looking . . . |
Ah, found it. This looks like a bug to me: msbuild/src/Tasks/Microsoft.Common.CurrentVersion.targets Lines 2701 to 2709 in 39e20dc
Nothing else makes that item transitive. This is triggered only when |
Success! If that's Okay, so you're saying Just in case you need it, by "the list" I'm referring to the one here which is what gets expanded from |
Correct.
That's a good question and will require more thought than I'm willing to put in at this time on Friday before a holiday weekend :) You don't have that many items, and we shouldn't take so long.
Have you ever seen it fail? I suspect it's not actually helpful but am not willing to commit either way at the moment.
|
I posted this and had a working resolution before the end of the workday. My expectations have already been exceeded! :D It'll take me some time to work through all of the other suggestions and I'll update when I get some more data. Thank you for the work so far! |
I wonder whether this could somehow be cheaply improved as well. Do these all have the same metadata count? |
They do not. It was different depending on which project it's referenced from and in our case there was a property called "NuGetPackageId" with the name of the referencing project, but I know there were more. In the brief look I took there were some items with a half dozen extra properties for the same ItemSpec. However, for this specific case (the de-duping logic) we could just use a custom comparer to override the get hash code logic since the hash set is ONLY used for the deduping (and immediately discarded) so there's no chance of bugs caused by the item metadata changing. That could be a perf improvement worth doing even if we fix the reference leaking problem. |
Okay, it was a little bit clunky but I was able to get it building with a build from the dev tree successfully. I deleted the output folder and built with
Yeah, this is definitely worth investigating more deeply. I dunno if there's anything that depends on those objects being output. Presumably not because you'd only see it when FindInvalidProjectReferences is enabled. |
Tried out using a hashcode that includes all object metadata instead of just ItemSpec: (warning: VERY janky code ahead) https://github.com/veleek/msbuild/pull/1/files?w=1. This is definitely faster, but took about 4 minutes. With the check disabled it takes about 1.5 minutes. I fiddled around with a few other things like caching the hashcodes so they were only ever generated once and that didn't seem to move the needle, so it's literally just the sheer quantity of items (46k) that increases the build time. With the check disabled, all these |
I looked at adding metadata into the hashcode but it's likely to be much more expensive in the general case as it has to resolve several dictionaries. I put up a PR with just a small change that still uses only itemspec. |
Yeah, I'm not sure it makes sense to include the metadata in the general case, most because it's not immutable (I think?). In this de-duping scenario the HashSet isn't persistent so mutability doesn't matter. just using a custom comparator to override GetHashCode means that the object metadata is traversed at most once for each item in the list (assuming no hash collisions which should be rare) so it'd be good enough and avoid calls to |
Issue Description
On large highly interdependent projects the
GetTargetPathWithTargetPlatformMoniker
andGetTargetPath
targets (from Microsoft.Common.CurrentVersion.targets) can end up taking multiple minutes to run.GetTargetPathWithTargetPlatformMonike
doesn't do anything expensive, it just creates a new project Item with some metadata, but it has aReturns
attrribute that returns@(TargetPathWithTargetPlatformMoniker)
which due to post processing in MSBuild to handle the returns, can take a long time to execute. Since theGetTargetPath
target is essentially a passthrough for this value, it takes almost the exact same amount of time. For our large project we have seen each of these targets take upwards of 3.5 minutes to run.The behavior can be seen to a lesser effect on other projects in the solution where each target takes ~20 second or ~1 minute to run.
Overall build time:
Time spent on the two targets (note: these targets are serial, not parallel, so it is almost minutes in total)
Steps to Reproduce
I'm happy to privately share a binlog of the build, but the descript here should be enough to identify the source of the problem.
Data
Call stack during 3minute hangs:
Analysis
Example project structure:
GetTargetPath
gets called on each of these, and because of the metadata introduced at different levels, it produces 7 differentTargetPathWithTargetPlatformMoniker
items. For Common.csproj, these items all have the sameItemSpec
(essentially just"Common.csproj"
) but they differ in their metadata.This example has 7 items. In our solution with 385 projects we have a PostBuild project that references every other project. When this runs, the
@(TargetPathWithTargetPlatformMoniker)
collection has ~50K items.As part of the post-processing for the target, since they have a
Returns
attribute, there is some work done to dedupe this collection. It uses aHashSet<TaskItem>
to do the deduping. However, the hashcode for the TaskItem only takes into account the ItemSpec so there ends up being a lot of hash collisions and it falls back to doing an expensive comparison which generates and compares the entire metadata collection of the item.Our PostBuild project is the most extreme example of this, but you can see that this problem appears even in projects that are significantly smaller, but this step still ends up taking an excessively long time (we have multiple projects that take >10s for each of these targets and it compounds over all the projects).
Versions & Configurations
We are using MSBuild 17.2.0, but i've analyzed the call paths and I don't see any changes between that version and the most recent build that would change this behavior.
Regression
Regression Details
We have been using this common PostBuild project for years, but apparently the slowdowns have only really started as part of modernizing to newer versions of MSBuild and .NET.
The text was updated successfully, but these errors were encountered: