Implement a NuGet-based SDK resolver #2850

jeffkl · 2018-01-10T06:22:38Z

An SDK resolver that activates only when a user has specified a version. This resolver is ordered to come before the DotNetSdkResolver but if a user does not specify a version (the default), then the NuGet SDK resolver is skipped.

The NuGet SDK resolver can be disabled with an environment variable just in case its causing issues or a user wants to override its functionality.

NuGet assemblies are loaded at runtime from their installed location. They either come come [VisualStudioInstallRoot\Common7\IDE\CommonExtensions\Microsoft\NuGet or next to MSBuild.exe. There is also an environment variable that can be set to specify a folder to use instead.

I've also placed references to NuGet into private subclasses so that assemblies are not loaded unless they are needed. This means that if you have no version specified and no global.json with versions, Newtonsoft.Json and NuGet.*.dll are not loaded. You only pay JIT penalty if you're using the functionality and hopefully we'll make up that time later in the build through caching.

Addresses item 6 in #2803
Related to #2278

jeffkl · 2018-01-10T06:34:49Z

FYI @nguerrera @emgarten @mishra14 @zhili1208

AndyGerlicher · 2018-01-10T15:45:52Z

src/Build/BackEnd/Components/SdkResolution/NuGet/NuGetSdkResolver.cs

+    {
+        public override string Name => nameof(NuGetSdkResolver);
+
+        public override int Priority => 2500;


To be explicit here for everyone reviewing, the DefaultSdkResolver is 10,000 and DotNetMSBuildSdkResolver is 5,000. So at 2,500 NuGetSdkResolver will run before any others (that we ship).

AndyGerlicher · 2018-01-10T16:07:30Z

src/Build/BackEnd/Components/SdkResolution/NuGet/NuGetSdkResolver.cs

+                            else
+                            {
+                                // This should never happen
+                                errors.Add("Package was supposed to be installed but wasn't");


Move the 2 error messages to resources please.

Oh and the message above.

I was having trouble with the wording, I can't see how the conditions would be hit in the first place. NuGet says it successfully restored the package but you have to do a secondary lookup of where it was put. Any suggestions for the messages?

Not concerned much with the message if it shouldn't ever happen. Maybe just Error resolving package '{0}'. NuGet restored successfully but the package was not found. Or something like that? Just think it should have the package name and be moved to resource.

rainersigwald

Many nits and questions, nothing very blocking.

I do want to see some internal tests set up to validate that this continues working in VS scenarios, with changing NuGet/Newtonsoft.Json versions as deployed. Don't want to find out about breakage too late.

rainersigwald · 2018-01-10T16:01:46Z

src/Build/Microsoft.Build.csproj

@@ -810,6 +819,7 @@
    <Compile Include="Evaluation\LazyItemEvaluator.EvaluatorData.cs" />
    <None Include="project.json" />
  </ItemGroup>
+  <ItemGroup />


nit: remove

rainersigwald · 2018-01-10T16:05:29Z

src/Build/project.json

@@ -3,6 +3,8 @@
    "net46": {
      "dependencies": {
        "Microsoft.VisualStudio.Setup.Configuration.Interop": "1.14.114",
+        "NuGet.Commands": "4.5.0",
+        "NuGet.Protocol": "4.5.0",


Do we need to get the NuGet team to add us to their automatically-drop-new-NuGet-assemblies PRs?

I'm not sure what this is?

They have a bot that creates PRs like dotnet/sdk#1851. I was wondering if we needed to get them to target us too. Hopefully not, since we don't need the very latest . . . until we adopt their version of the resolverEx thing.

rainersigwald · 2018-01-10T16:06:26Z

src/Build/BackEnd/Components/SdkResolution/NuGet/NuGetSdkResolverBase.cs

+        /// <summary>
+        /// The sub-folder under the Visual Studio installation where the NuGet assemblies are located.
+        /// </summary>
+        public const string PathToNuGetUnderVisualStudioRoot = @"Common7\IDE\CommonExtensions\Microsoft\NuGet";


Is this documented? How will we know if it changes?

I have no idea, maybe @emgarten knows?

This is considered a well known location and it is defined in the VSIX manifest, we have it set here also: https://github.com/NuGet/NuGet.BuildTasks/blob/1d8af3499f94a32c1d128a42faceade39c1f4337/src/Microsoft.NuGet.Build.Tasks/ImportBeforeAfter/Microsoft.NuGet.ImportAfter.targets#L14

I would not expect this to change, if it did it would be very hard to find where it moved to.

rainersigwald · 2018-01-10T16:08:42Z

src/Build/BackEnd/Components/SdkResolution/NuGet/NuGetSdkResolverBase.cs

+#elif !CLR2COMPATIBILITY
+                    return Assembly.UnsafeLoadFrom(assemblyPath);
+#else
+                    return Assembly.LoadFrom(assemblyPath);


How do these loads work in the face of deployed NuGet DLLs that are a different version from what we built against? Is it different between Core and full framework?

They are effectively assembly binding redirects. If we compile against 4.3 we can safely load 4.5 instead. As long as there are no breaking API changes, it should work.

rainersigwald · 2018-01-10T16:09:42Z

src/Build/project.json

@@ -3,6 +3,8 @@
    "net46": {
      "dependencies": {
        "Microsoft.VisualStudio.Setup.Configuration.Interop": "1.14.114",
+        "NuGet.Commands": "4.5.0",
+        "NuGet.Protocol": "4.5.0",


Is the Newtonsoft.Json reference coming to us transitively through this? Since we use it explicitly I'd like to see it mentioned explicitly.

Yes, it comes through NuGet.ProjectModel

Technically the version doesn't matter since we only load it from where NuGet is at. We only have one call which I highly doubt would ever be broken. But if you feel strongly about it...

rainersigwald · 2018-01-10T16:47:15Z

src/Build/BackEnd/Components/SdkResolution/NuGet/RestoreRunnerEx.cs

+        public static Task<IReadOnlyList<RestoreResultPair>> RunWithoutCommit(string projectPath, string id, string version, ISettings settings, ILogger logger)
+        {
+            // NuGet requires at least one framework, we use .NET Standard here just to get the API to do work.  The framework is not actually used.
+            List<NuGetFramework> targetFrameworks = new List<NuGetFramework>


make this a static readonly field?

rainersigwald · 2018-01-10T16:50:38Z

src/Build/BackEnd/Components/SdkResolution/SdkResolverLoader.cs

-            if (potentialResolvers.Count == 0) return resolvers;
+            if (potentialResolvers.Count == 0)
+            {
+                return resolvers.Count == 1 ? resolvers : resolvers.OrderBy(t => t.Priority).ToList();


Was this optimization noticeably faster?

I left it in to keep existing functionality. @AndyGerlicher had the optimization in there first, possibly by accident

rainersigwald · 2018-01-10T16:51:46Z

src/Build/BackEnd/Components/SdkResolution/SdkResolverService.cs

+                        {
+                            result = (SdkResult) sdkResolver.Resolve(sdk, context, resultFactory);
+                        }
+                        catch (Exception e) when (sdkResolver is NuGetSdkResolver && e is FileNotFoundException)


Oof, this is specific. But I can't think offhand of a better way.

Yeah we went back and forth on this. We're explicitly adding the resolver so we felt comfortable with the special case here. I'll add a comment to explain the reasoning

rainersigwald · 2018-01-10T16:52:31Z

src/Build/BackEnd/Components/SdkResolution/SdkResolverService.cs

+                        catch(Exception e)
+                        {
+                            // MSB4242: The SDK resolver "{0}" failed to run. {1}
+                            loggingContext.LogWarning(null, new BuildEventFileInfo(sdkReferenceLocation), "CouldNotRunSdkResolver", sdkResolver.Name, e.Message);


What were we doing for this before? Is it ok that this is no longer fatal?

Yes before it was a fatal error. When testing things out, a fatal build error when something was wrong with a resolver seemed like overkill. What do you think?

rainersigwald · 2018-01-10T16:53:07Z

src/Build/project.json

@@ -14,19 +16,26 @@
    "netstandard1.5": {
      "dependencies": {
        "NETStandard.Library": "1.6.0",
+        "NuGet.Commands": "4.5.0",
+        "NuGet.Protocol": "4.5.0",


nit: since these are the same across frameworks it'd be nice to have them in a common node.

AndyGerlicher

Looks good once Rainer's comments are addressed!

jeffkl · 2018-01-10T17:41:02Z

src/Build/BackEnd/Components/SdkResolution/SdkResolverService.cs

+                        catch(Exception e)
+                        {
+                            // MSB4242: The SDK resolver "{0}" failed to run. {1}
+                            loggingContext.LogWarning(null, new BuildEventFileInfo(sdkReferenceLocation), "CouldNotRunSdkResolver", sdkResolver.Name, e.Message);


Yes before it was a fatal error. When testing things out, a fatal build error when something was wrong with a resolver seemed like overkill. What do you think?

jeffkl · 2018-01-10T17:42:08Z

src/Build/project.json

@@ -3,6 +3,8 @@
    "net46": {
      "dependencies": {
        "Microsoft.VisualStudio.Setup.Configuration.Interop": "1.14.114",
+        "NuGet.Commands": "4.5.0",
+        "NuGet.Protocol": "4.5.0",


I'm not sure what this is?

jeffkl · 2018-01-10T17:44:27Z

src/Build/project.json

@@ -3,6 +3,8 @@
    "net46": {
      "dependencies": {
        "Microsoft.VisualStudio.Setup.Configuration.Interop": "1.14.114",
+        "NuGet.Commands": "4.5.0",
+        "NuGet.Protocol": "4.5.0",


Yes, it comes through NuGet.ProjectModel

Technically the version doesn't matter since we only load it from where NuGet is at. We only have one call which I highly doubt would ever be broken. But if you feel strongly about it...

jeffkl · 2018-01-10T17:44:41Z

src/Build/project.json

@@ -14,19 +16,26 @@
    "netstandard1.5": {
      "dependencies": {
        "NETStandard.Library": "1.6.0",
+        "NuGet.Commands": "4.5.0",
+        "NuGet.Protocol": "4.5.0",


jeffkl · 2018-01-10T17:45:06Z

src/Build/project.json

        "System.Collections.Immutable": "1.2.0",
        "System.Collections.NonGeneric": "4.0.1",
        "System.Diagnostics.Contracts": "4.0.1",
        "System.Diagnostics.FileVersionInfo": "4.0.0",
        "System.Diagnostics.Process": "4.1.0",
        "System.Diagnostics.TraceSource": "4.0.0",
+        "System.IO": "4.1.0",


These had to be added explicitly because the reference to NuGet caused some unwanted upgrades.

cdmihai · 2018-01-10T19:34:02Z

src/Build/BackEnd/Components/SdkResolution/NuGet/NuGetSdkResolver.cs

+
+        public override SdkResultBase Resolve(SdkReference sdk, SdkResolverContextBase context, SdkResultFactoryBase factory)
+        {
+            object parsedSdkVersion;


Why object?

Ah, comment in the nested class :)

In reply to: 160777249 [](ancestors = 160777249)

jnm2 · 2018-01-25T18:08:17Z

I've been waiting to hear when I can write a build script that works without requiring any specific machine state (think pristine Windows installation) by pulling MSBuild and all SDKs from NuGet. I'm specifically interested in the legacy .csproj, the new .csproj, and .vcxproj SDKs.

@rainersigwald told me to wait for #1493 which has been inactive for eight months. Does this PR bring us any closer to that? Is progress being made and I'm just still out of the loop?

rainersigwald · 2018-01-25T18:26:32Z

@jnm2 Thanks for pointing out that that issue was super stale; I updated it a bit.

This improves the situation for custom SDKs, which can now be acquired from NuGet.

I believe there is no planned work to make the .NET Core SDK available via NuGet, so you must still deploy it separately (or get MSBuild through the Core SDK, which is a single unzippable entity).

There's no planned work to make the installed C# or C++ targets available on NuGet, which doesn't change based on this--they do not use MSBuild Sdks so Sdk acquisition doesn't affect those scenarios.

jnm2 · 2018-01-25T18:43:34Z

Thank you!

There's no planned work to make the installed C# or C++ targets available on NuGet, which doesn't change based on this--they do not use MSBuild Sdks so Sdk acquisition doesn't affect those scenarios.

Who do I talk to?

jeffkl added 3 commits January 9, 2018 22:15

Implementation

a7faf14

Fix sorting of resolvers

45841e3

Fix build warning

77493d0

jeffkl mentioned this pull request Jan 10, 2018

NuGet-based SDK resolver design spec #2803

Closed

7 tasks

Fix missing return statement and other minor issue

831d754

AndyGerlicher reviewed Jan 10, 2018

View reviewed changes

jeffkl added 2 commits January 10, 2018 08:13

Add string for Resolving SDK 'X" message

da1a83d

Add two error message strings and better comment

c2020ad

rainersigwald approved these changes Jan 10, 2018

View reviewed changes

AndyGerlicher approved these changes Jan 10, 2018

View reviewed changes

jeffkl commented Jan 10, 2018

View reviewed changes

Address comments

d0e8ac7

jeffkl merged commit a955af8 into dotnet:master Jan 10, 2018

jeffkl deleted the NuGetSdkResolver branch January 10, 2018 19:08

cdmihai reviewed Jan 10, 2018

View reviewed changes

jeffkl mentioned this pull request Jan 10, 2018

Minor improvements to SDK resolver execution #2851

Merged

AndyGerlicher mentioned this pull request Jan 16, 2018

MSBuild 15.6.54 dotnet/cli#8396

Merged

rainersigwald mentioned this pull request Jan 25, 2018

MSBuild 15 Sdk Design #1493

Closed

13 tasks

zarenner mentioned this pull request Jun 11, 2019

dotnet msbuild fails to restore from private Azure feed #4430

Open

Implement a NuGet-based SDK resolver #2850

Implement a NuGet-based SDK resolver #2850

Conversation

jeffkl commented Jan 10, 2018 • edited Loading

jeffkl commented Jan 10, 2018

Choose a reason for hiding this comment

AndyGerlicher Jan 10, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rainersigwald left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AndyGerlicher left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jnm2 commented Jan 25, 2018 • edited Loading

rainersigwald commented Jan 25, 2018

jnm2 commented Jan 25, 2018

jeffkl commented Jan 10, 2018 •

edited

Loading

AndyGerlicher Jan 10, 2018 •

edited

Loading

jnm2 commented Jan 25, 2018 •

edited

Loading