.NET 8: Generate report of non-explicit dependencies for each repo in source build

[mmitche]

Description

For #2980, we need a list that repos can drive off. They need to be able to make the following decisions:

• Does the dependency get added to eng/Version.Details.xml?
• If it does, does the dependency get automatic updates? If so, add them. If not, pin and provide justification.

The list is would be populated with the following:

• Properties in the eng/Versions.props of the format <Foo>Version or <Foo>PackageVersion and
• That do not already map to a corresponding eng/Version.Details.xml element and
• And which are produced by a repository that is built earlier in source-build.

[mmitche]

@MichaelSimons I think this could be added in the current tarball generation phase (eventually the VMR update). Thoughts/objections?

[MichaelSimons]

I have no objection to adding it. If it impacts the tarball generation speed by a non-trivial amount I would want it to be optional.

[mmitche]

Should be very quick. Just a scan of the version files in each repo.

[mmitche]

And probably a comparison against a baseline.

[mmitche]

@MichaelSimons I had a chat with @crummel about this and I think this work does not add significant value, and generally solves itself with:

• PVP flow in product source build
• Pre-built detection turned on in repo source build legs.

The logic goes like this:

Currently, repo level source build does not generate a package version props for the repo given the input packages available in the source build intermediates. Instead, the eng/PackageVersions.props is used, and the intermediate package is used to provide dependencies. eng/PackageVersions.props is updated via Maestro dependency flow, so it's "pvp" style flow already. This means that undeclared dependencies do not get updated in eng/PackageVersions.props and will instead show up as pre-builts in the repo level source build. The "Microsoft" and source build methodologies are closely matched at the repo level.

Product level source-build doesn't currently do PVP flow. When it switches to PVP flow, product level source build will begin to match repo level source build. PVP flow acts as a "mini Maestro". Undeclared dependencies will show up as prebuilts.

Thus, once prebuilt detection is turned on:

• Undeclared dependencies are caught at the repo level
• Undeclared dependencies are caught at the product level.

The one hole in this is that "non-flowing dependencies" would not be caught. If, for instance aspnet had an active subscription to arcade but efcore did not, repo level source build would succeed for both repos, but product level source build would supply a newer arcade than efcore was validated with at the repo level. This case isn't fundamentally different than any other coherency issue though, and a subscription isn't a guarantee that a dependency is getting updated regularly. To me that needs to be handled by a different mechanism or monitor.

I suggest we close this, and work on PVP flow ASAP. I can take that work if you'd like.

[MichaelSimons]

I suggest we close this, and work on PVP flow ASAP. I can take that work if you'd like.

Agreed. It would be great it you can pickup the PVP work now. I will assign the issue to you.