Proposal: implement a SdkResolver for NuGet packages

[natemcmaster]

MSBuild 15.3 includes an new API, Microsoft.Build.Framework.SdkResolver. (see dotnet/msbuild#2002). In theory, we could implement a NuGetPackageSdkResolver that resolves the "Sdk" property to files in the NuGet cache.

Goal
Distribute versioned MSBuild tasks, targets, and other tooling as a NuGet package.

Usage

<Project>
   <Sdk Name="Microsoft.NET.Sdk" />
   <Sdk Name="Microsoft.NET.Test.Sdk" MinimumVersion="15.0.0" />
   <Sdk Name="my.team.shared.buildsettings" Version="1.0.0-*" />
</Project>

In this example, Microsoft.Net.Sdk is not a nuget package, so the resolver would no-op.
On the other hand, Microsoft.NET.Test.Sdk and my.team.shared.buildsettings are nuget packages that contains MSbuild targets and tasks. The resolver could download these packages to make them available to MSBuild.

Layout of an "SDK" package

(pkg root)/
  - build/
    - Sdk/
      + Sdk.props
      + Sdk.targets

What the resolver does
(pseudocode)

class NuGetSdkResolver : Microsoft.Build.Framework.SdkResolver
{
    public override SdkResult Resolve(SdkReference sdk, SdkResolverContext context, SdkResultFactory factory)
    {
        var resolvedVersion = ChoosePackageVersionOrDefault(sdk.Version ?? sdk.MinimumVersion);
        RestorePackageIfNotFound(sdk.Name, resolvedVersion);
        var sdkPath = Path.Combine("%NUGET_PACKAGES%", sdk.Name, resolvedVersion, "build/Sdk");

        return factory.IndicateSuccess(sdkPath, resolvedVersion);
    }
}

Why not just use a PackageReference?
Many simple scenarios are already solved by adding build assets to a package, however, MSBuild tooling from an NuGet package still has some limitations:

• Build requires two msbuild invocations. msbuild /t:Restore && msbuild /t:Build. Just running msbuild /t:Restore;Build causes issues because nuget.g.targets and UsingTasks are not resolved soon enough.
• You have to express TargetFramework to make PackageReference work. This property only makes sense for csproj, but doesn't have meaning for MSBuild scripts that do other build automation.
• You cannot have a PackageReference that conditionally adds more PackageReferences or DotNetCliToolReference.
• Users have to know to set PrivateAssets=All to avoid having tooling packages end up in the generated nuspec.
• You have to import Microsoft.Common.targets (and everything it brings with it) in order to use PackageReference.

One possibility
If this were implemented, I could easily create a package that looks like this:

Microsoft.AspNetCore.LTS.Lineup.nupkg/
- build
  - Sdk/
     Sdk.targets
     Tasks.dll

<!-- Sdk.targets -->
<Project>
  <UsingTask AssemblyFile="Tasks.dll" TaskName="CustomAspNet.Build.Task" />
  <ItemGroup>
     <!-- Update package references to the preferred lineup version if they are included in the project.-->
     <PackageReference Update="Kestrel" Version="1.0.4" IsImplicitlyDefined="true" />
     <PackageReference Update="Mvc" Version="1.0.5" IsImplicitlyDefined="true"  />
     <!-- add a CLI tool -->
     <DotNetCliToolReference Include="dotnet-watch" Version="1.0.1" />
   </ItemGroup>
</Project>

This could solve the "Lineups" feature. #2572

cc @emgarten @AndyGerlicher @AArnott @nguerrera

[emgarten]

This looks like a great idea! I've been looking for a good extension point for tool packages that need to participate in restore such as git versioning packages. Currently NuGet excludes these imports during restore since they can cause the 1st and 2nd restore results to differ. If this ran before hand it would solve the issue.

Lineups is an interesting scenario here, this could work to override and set a specific set of packages. There are a few other parts to lineup though such as locking the the currently selected packages down for repeatable builds.

[emgarten]

I'll try a few things with SdkResolver and see how it goes. Do the items need to be named SDK? NuGet would need a good way to determine if these are actually packages or not since it would need to attempt finding these from all sources.

[natemcmaster]

There are some quirks here to think, through, such as deciding how to resolve the package version to use (i.e. when no version is provided) and making SDK lookup fast despite the need to hit the network.

Do the items need to be named SDK?

The samples above were just one syntax. The value for SDK can come from an attribute on Project and Import as well:

<Project Sdk="Community.FSharp.Sdk">

<Import Project="Sdk.props" Sdk="Microsoft.NET.Sdk" />
<Import Project="Sdk.props" Sdk="Microsoft.NET.Sdk/2.0.0" />
<Import Project="Sdk.props" Sdk="Microsoft.NET.Sdk/min=1.0.0" />

[AArnott]

This sounds very interesting, because both as @emgarten said for git versioning packages and also for real SDKs.
In your description, @natemcmaster, you explicitly excluded Microsoft.NET.Sdk as a potential example of an SDK acquired from NuGet. But why couldn't it come from NuGet? Honestly that's what gets me the most jazzed about your proposal here. I want all my build dependencies to come from NuGet (except msbuild.exe itself, due to the chicken-and-egg problem), including all tasks and .targets. Is there something fundamentally limiting to prevent the original SDK from coming from nuget in your proposal?

[natemcmaster]

Early versions of the Microsoft.NET.SDK actually were NuGet packages. There's an issue on the backlog to make it available via NuGet -- dotnet/sdk#300 --
But there are some features that would be harder to implement, like the templates, bundled versioning of the runtime, and some of the "dotnet command" functions that aren't actually msbuild targets.

Btw, I think this resolver would help with some of the ideas you brought up in dotnet/msbuild#1756

[rainersigwald]

Please don't reinvent this! This is the entire goal of MSBuild SDKs. We're making incremental progress in conjunction with the SDK team (and, I thought, the NuGet team, but this issue reveals that that understanding was wrong).

cc @AndyGerlicher @Microsoft/msbuild-maintainers

[natemcmaster]

Can clarify what you're working on with the SDK team? The main thing I'm after is a way to install targets/tasks from arbitrary NuGet packages. PackageReference is not quite good enough for reasons listed above, so I opened this issue on NuGet as a way to improve the acquisition story for MSBuild targets/tasks. Is there something else in the works that would overlap with this?

[rainersigwald]

Just talked about this in MSBuild/SDK/NuGet sync today.

This was cut from 15.3 planning primarily because of concerns about doing a blocking network call during project evaluation.

Other concerns:

• Progress reporting and network I/O during resolve.
• Locking around package downloads (from multiple projects that refer to the same SDK package).
• Version unification and closure computation.
• Lineups--seem like they might be necessary but aren't designed. 📝 There is a concept of "lineup" for PackageReferences and a lineup for SDKs that are probably distinct.
• Referencing SDKs across resolvers (especially while the core SDK isn't delivered by package).

cc @rrelyea

[rrelyea]

Consider and discuss post 15.3. As @rainersigwald said, this is not in plan for 15.3.

[rainersigwald]

MSBuild SDK spec, with many mentions of a NuGet resolver: dotnet/msbuild#1493.

[clairernovotny]

My ask when designing this feature is that it's not tied to .NET or TFM's. That is, it should ideally support any project type that uses CPS to enable it to have its own SDK. The example I'm thinking of right now is Rust, but it can apply to any other language/platform.

[jp2masa]

I'm working on a project system based on CPS, and I'd like to distribute the build system as an Sdk (ideally a package Sdk). I can't use PackageReference because MSBuild seems to expect for at least one import for props and one for targets. Is this planned for the next VS/MSBuild/NuGet release? Also, is it possible to implement a custom Sdk resolver in my project system, without installing it to the MSBuild installation folder?

[jeffkl]

I am working with the NuGet team to add this in the next release of MSBuild.

https://github.com/jeffkl/NuGetSdkResolver

[natemcmaster]

Also, is it possible to implement a custom Sdk resolver in my project system, without installing it to the MSBuild installation folder?

I hit the same limitation too.
dotnet/msbuild#2278 So far I haven't seen a response to this from the MSBuild team.

[japj]

I was wondering, if I want to load my own SDK without having to code my own resolver, would it be possible to extend the MSBuildSDKsPath mechanism to support multiple paths? Then on my build I can just 'append' to the current path.
Or is that already supported with a different mechanism? (My SDK would be in source control and not be installed in the MSBuild sdk folder)

[japj]

Sorry, just realised this is the nuget repo.. not the MSBuild/SDK one

[mishra14]

Closing in favor of dotnet/msbuild#2278 and https://github.com/jeffkl/NuGetSdkResolver.

We are working with msbuild folks on this. On our side we are tracking #5919 for a download API that might be consumed by msbuild folks.