Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The dotnet store is incorrectly including BCL assemblies when targeting .NET Core 3.1 #10973

Closed
normj opened this issue Mar 24, 2020 · 43 comments
Closed

The dotnet store is incorrectly including BCL assemblies when targeting .NET Core 3.1 #10973

normj opened this issue Mar 24, 2020 · 43 comments
Assignees
@dsplaisted

Comments

@normj
Copy link
Contributor

normj commented Mar 24, 2020 •
edited
Loading

Using the following store manifest that only includes Newtonsoft.Json

<Project Sdk="Microsoft.NET.Sdk">
  <ItemGroup>
    <PackageReference Include="Newtonsoft.Json" Version="9.0.1" />
  </ItemGroup>
</Project>

I execute the following command to create a store for .NET Core 2.1:

dotnet store --manifest .\manifest.xml -f netcoreapp2.1 -o ./store2.1 -r linux-x64 --skip-optimization

the store artifact.xml contains only Newtonsoft.Json as expected.

<StoreArtifacts>
  <Package Id="Newtonsoft.Json" Version="9.0.1" />
</StoreArtifacts>

But then creating the store for .NET Core 3.1 using the command:

dotnet store --manifest .\manifest.xml -f netcoreapp3.1 -o ./store3.1 -r linux-x64 --skip-optimization

Creates a store with a lot of runtime and system dlls which bloat the package store and also those assembles fail to the store optimization.

<StoreArtifacts>
  <Package Id="Microsoft.CSharp" Version="4.0.1" />
  <Package Id="Newtonsoft.Json" Version="9.0.1" />
  <Package Id="runtime.any.System.Collections" Version="4.0.11" />
  <Package Id="runtime.any.System.Diagnostics.Tools" Version="4.0.1" />
  <Package Id="runtime.any.System.Globalization" Version="4.0.11" />
  <Package Id="runtime.any.System.IO" Version="4.1.0" />
  <Package Id="runtime.any.System.Reflection" Version="4.1.0" />
  <Package Id="runtime.any.System.Reflection.Extensions" Version="4.0.1" />
  <Package Id="runtime.any.System.Reflection.Primitives" Version="4.0.1" />
  <Package Id="runtime.any.System.Resources.ResourceManager" Version="4.0.1" />
  <Package Id="runtime.any.System.Runtime" Version="4.1.0" />
  <Package Id="runtime.any.System.Runtime.Handles" Version="4.0.1" />
  <Package Id="runtime.any.System.Runtime.InteropServices" Version="4.1.0" />
  <Package Id="runtime.any.System.Text.Encoding" Version="4.0.11" />
  <Package Id="runtime.any.System.Text.Encoding.Extensions" Version="4.0.11" />
  <Package Id="runtime.any.System.Threading.Tasks" Version="4.0.11" />
  <Package Id="runtime.unix.System.Diagnostics.Debug" Version="4.0.11" />
  <Package Id="runtime.unix.System.IO.FileSystem" Version="4.0.1" />
  <Package Id="runtime.unix.System.Private.Uri" Version="4.0.1" />
  <Package Id="runtime.unix.System.Runtime.Extensions" Version="4.1.0" />
  <Package Id="System.Dynamic.Runtime" Version="4.0.11" />
  <Package Id="System.IO.FileSystem.Primitives" Version="4.0.1" />
  <Package Id="System.Linq" Version="4.1.0" />
  <Package Id="System.Linq.Expressions" Version="4.1.0" />
  <Package Id="System.ObjectModel" Version="4.0.12" />
  <Package Id="System.Reflection.Emit" Version="4.0.1" />
  <Package Id="System.Reflection.Emit.ILGeneration" Version="4.0.1" />
  <Package Id="System.Reflection.Emit.Lightweight" Version="4.0.1" />
  <Package Id="System.Reflection.TypeExtensions" Version="4.1.0" />
  <Package Id="System.Runtime.Serialization.Primitives" Version="4.1.1" />
  <Package Id="System.Text.RegularExpressions" Version="4.1.0" />
  <Package Id="System.Threading" Version="4.0.11" />
  <Package Id="System.Threading.Tasks.Extensions" Version="4.0.0" />
  <Package Id="System.Xml.ReaderWriter" Version="4.0.11" />
  <Package Id="System.Xml.XDocument" Version="4.0.11" />
</StoreArtifacts>

This change in behavior seems to make the dotnet store command a broken experience.

@nguerrera We have talked about dotnet store in the past. Can you shed any light on why this is happening now?
This was referenced Mar 24, 2020
Closed
Merged
@normj
Copy link
Contributor Author

normj commented Mar 31, 2020

Friendly ping on this to get some attention. This is breaking some of our scenarios for .NET Core 3.1.

@rpopovych
Copy link

rpopovych commented Apr 3, 2020 •
edited
Loading

This is a bummer. Prevents us from using 3.1 in some of our AWS Lambdas. Hoping there is an easy resolution/workaround for this issue.

@davidruhmann
Copy link

davidruhmann commented Apr 10, 2020 •
edited
Loading

Wonder if the root cause is related to this issue dotnet/runtime#2836;

Found the issue referenced in the code to handle an additional file being stored. https://github.com/dotnet/sdk/blob/master/src/Tests/Microsoft.NET.Publish.Tests/GivenThatWeWantToStoreAProjectWithDependencies.cs#L92

From a quick look at different versions, this only seems to happen for Newtonsoft.Json versions < 11. Only major change that I initially see to cause this is that version 11 starts referencing netstandard2.0.

@dsplaisted
Copy link
Member

This is a result of changes we made in .NET Core 3. We no longer use PackageReferences to represent .NET Core Framework. This is discussed here: dotnet/designs#50

The old version of NewtonSoft.Json has dependencies to packages which are part of .NET Core. Because we don't represent .NET Core as a PackageReference anymore, the dotnet store logic no longer knows to exclude them from the store.

The ideal fix to this would likely be NuGet/Home#7344, and possibly additional changes in dotnet store to use it.

Possible workarounds:

  • Use a later version of NewtonSoft.Json (or in general, use packages that don't have these older dependencies, which will generally be the case if the package was built for .NET Core 2.0 or .NET Standard 2.0 or higher).
  • Exclude each unneeded package from the manifest by adding the following
  <Target Name="AddPackagesToPrune" AfterTargets="PrepforRestoreForComposeStore">
    <ItemGroup>
      <PackagesToPrune Include="Microsoft.CSharp" />
      <PackagesToPrune Include="System.Collections" />
      <PackagesToPrune Include="System.Diagnostics.Debug" />
      <PackagesToPrune Include="System.Dynamic.Runtime" />
      <PackagesToPrune Include="System.Globalization" />
      <PackagesToPrune Include="System.IO" />
      <PackagesToPrune Include="System.Linq" />
      <PackagesToPrune Include="System.Linq.Expressions" />
      <PackagesToPrune Include="System.ObjectModel" />
      <PackagesToPrune Include="System.Reflection" />
      <PackagesToPrune Include="System.Reflection.Extensions" />
      <PackagesToPrune Include="System.Resources.ResourceManager" />
      <PackagesToPrune Include="System.Runtime" />
      <PackagesToPrune Include="System.Runtime.Extensions" />
      <PackagesToPrune Include="System.Runtime.Serialization.Primitives" />
      <PackagesToPrune Include="System.Text.Encoding" />
      <PackagesToPrune Include="System.Text.Encoding.Extensions" />
      <PackagesToPrune Include="System.Text.RegularExpressions" />
      <PackagesToPrune Include="System.Threading" />
      <PackagesToPrune Include="System.Threading.Tasks" />
      <PackagesToPrune Include="System.Xml.ReaderWriter" />
      <PackagesToPrune Include="System.Xml.XDocument" />
    </ItemGroup>
    <PropertyGroup>
      <PackagesToPrune>$(PackagesToPrune);@(PackagesToPrune)</PackagesToPrune>
    </PropertyGroup>
  </Target>

Note that if there is a PackageToPrune which is not part of the dependency graph, you will get a NullReferenceException from one of the tasks.

@normj
Copy link
Contributor Author

normj commented Apr 15, 2020

Interesting. In my case dotnet store is used as a general tool and I don't have control what packages are in the manifest. I happen to pick Newtonsoft.Json as an example because one of my libraries has a dependency on that version that is commonly put into the store along with the customers packages.

I have the option to make a modifications to a copy of the inputted manifest and use that when calling dotnet store. I need to come up with a way given a manifest be able to compute what packages to prune or at a minimum detect that package will bring in framework packages and they should upgrade their package versions.

@dsplaisted
Copy link
Member

@normj Could you update your library which depends on NewtonSoft.Json to depend on at least 11.0.1? That should help mitigate the issue.

@normj
Copy link
Contributor Author

normj commented Apr 22, 2020

I can, but I can't control whether somebody uses our tool that relies dotnet store with dependencies that will also have this problem. Is there anyway for me to programmatically detect when a store manifest would potentially bring in these bad refs? I'm afraid I would need to build the whole dependency graph myself first and inspect before calling dotnet store.

@davidruhmann
Copy link

@dsplaisted, do you see any issues with this type of workaround until NuGet/Home#7344 is resolved?

  • Run dotnet store on the initial manifest.
  • Create a new manifest file with a prune list (consisting of entries of dependencies in the artifact.xml that also exist in the runtime folder (e.g. /usr/local/share/dotnet/shared/Microsoft.NETCore.App/3.1.3)
    • from System.Runtime.InteropServices.RuntimeEnvironment.GetRuntimeDirectory() but for the target framework.
  • Delete the old output.
  • Then run dotnet store again on the new manifest file.

Am I missing any major aspect regarding this?

@dsplaisted
Copy link
Member

@davidruhmann I think something like that could work, but mapping from the dependencies in the runtime folder to packages that should be excluded may not be simple. You have to consider not just the file name but the version number, and also map it to the right NuGet package.

@davidruhmann
Copy link

Appreciate the quick reply!

Version did cross my mind, but wouldn't that just be for higher versioned references? Which, if I understand correctly, could still be resolved by the store consumer? Meaning the store would possibly be missing some elements, but not the flip side of including unnecessary elements.

As for the naming of nuget packages to BCL components, are there many differences?

I am thinking of this workaround in a scope of 80% success or something like that. If it can solve the majority of conflicts for the current time until NuGet/Home#7344 is complete; that should be good enough vs making the workaround too complex. Even if the other 20% result in failure, the original bloated store could still be used or fail the process entirely saying unsupported.

@clearwaterstream
Copy link

@davidruhmann I must admit I have not patience for gardening, but pruning I did try. Initial manifest -> dotnet store -> look at resulting artifact.xml -> add the extraneous packages to prune list on initial manifest -> dotnet store again.

What I found is that the AddPackagesToPrune would choke on certain packages. Let's do a simple example.

Initial manifest:

<Project Sdk="Microsoft.NET.Sdk.Web">
  <ItemGroup>
	<PackageReference Include="Microsoft.AspNetCore.Authentication.JwtBearer" Version="3.1.3" />
  </ItemGroup>
</Project>

dotnet store --manifest initial.xml -f netcoreapp3.1 -o bin/dotnetcore/store -r linux-x64 --skip-optimization

Output is:

<StoreArtifacts>
  <Package Id="Microsoft.AspNetCore.Authentication.JwtBearer" Version="3.1.3" />
  <Package Id="Microsoft.CSharp" Version="4.3.0" />
  <Package Id="Microsoft.IdentityModel.JsonWebTokens" Version="5.5.0" />
  <Package Id="Microsoft.IdentityModel.Logging" Version="5.5.0" />
  <Package Id="Microsoft.IdentityModel.Protocols" Version="5.5.0" />
  <Package Id="Microsoft.IdentityModel.Protocols.OpenIdConnect" Version="5.5.0" />
  <Package Id="Microsoft.IdentityModel.Tokens" Version="5.5.0" />
  <Package Id="Newtonsoft.Json" Version="10.0.1" />
  <Package Id="runtime.any.System.Collections" Version="4.3.0" />
  <Package Id="runtime.any.System.Diagnostics.Tools" Version="4.3.0" />
  <Package Id="runtime.any.System.Diagnostics.Tracing" Version="4.3.0" />
  <Package Id="runtime.any.System.Globalization" Version="4.3.0" />
  <Package Id="runtime.any.System.IO" Version="4.3.0" />
  <Package Id="runtime.any.System.Reflection" Version="4.3.0" />
  <Package Id="runtime.any.System.Reflection.Extensions" Version="4.3.0" />
  <Package Id="runtime.any.System.Reflection.Primitives" Version="4.3.0" />
  <Package Id="runtime.any.System.Resources.ResourceManager" Version="4.3.0" />
  <Package Id="runtime.any.System.Runtime" Version="4.3.0" />
  <Package Id="runtime.any.System.Runtime.Handles" Version="4.3.0" />
  <Package Id="runtime.any.System.Runtime.InteropServices" Version="4.3.0" />
  <Package Id="runtime.any.System.Text.Encoding" Version="4.3.0" />
  <Package Id="runtime.any.System.Text.Encoding.Extensions" Version="4.3.0" />
  <Package Id="runtime.any.System.Threading.Tasks" Version="4.3.0" />
  <Package Id="runtime.unix.System.Diagnostics.Debug" Version="4.3.0" />
  <Package Id="runtime.unix.System.IO.FileSystem" Version="4.3.0" />
  <Package Id="runtime.unix.System.Private.Uri" Version="4.3.0" />
  <Package Id="runtime.unix.System.Runtime.Extensions" Version="4.3.0" />
  <Package Id="System.Buffers" Version="4.3.0" />
  <Package Id="System.Collections.NonGeneric" Version="4.3.0" />
  <Package Id="System.Collections.Specialized" Version="4.3.0" />
  <Package Id="System.ComponentModel" Version="4.3.0" />
  <Package Id="System.ComponentModel.Primitives" Version="4.3.0" />
  <Package Id="System.ComponentModel.TypeConverter" Version="4.3.0" />
  <Package Id="System.Dynamic.Runtime" Version="4.3.0" />
  <Package Id="System.Globalization.Extensions" Version="4.3.0" />
  <Package Id="System.IdentityModel.Tokens.Jwt" Version="5.5.0" />
  <Package Id="System.IO.FileSystem.Primitives" Version="4.3.0" />
  <Package Id="System.Linq" Version="4.3.0" />
  <Package Id="System.Linq.Expressions" Version="4.3.0" />
  <Package Id="System.ObjectModel" Version="4.3.0" />
  <Package Id="System.Reflection.Emit" Version="4.3.0" />
  <Package Id="System.Reflection.Emit.ILGeneration" Version="4.3.0" />
  <Package Id="System.Reflection.Emit.Lightweight" Version="4.3.0" />
  <Package Id="System.Reflection.TypeExtensions" Version="4.3.0" />
  <Package Id="System.Runtime.Numerics" Version="4.3.0" />
  <Package Id="System.Runtime.Serialization.Formatters" Version="4.3.0" />
  <Package Id="System.Runtime.Serialization.Primitives" Version="4.3.0" />
  <Package Id="System.Security.Cryptography.Cng" Version="4.5.0" />
  <Package Id="System.Text.RegularExpressions" Version="4.3.0" />
  <Package Id="System.Threading" Version="4.3.0" />
  <Package Id="System.Threading.Tasks.Extensions" Version="4.3.0" />
  <Package Id="System.Xml.ReaderWriter" Version="4.3.0" />
  <Package Id="System.Xml.XDocument" Version="4.3.0" />
  <Package Id="System.Xml.XmlDocument" Version="4.3.0" />
</StoreArtifacts>

initial.xml with one package added to prune:

<Project Sdk="Microsoft.NET.Sdk.Web">
  <ItemGroup>
	<PackageReference Include="Microsoft.AspNetCore.Authentication.JwtBearer" Version="3.1.3" />
  </ItemGroup>
  <Target Name="AddPackagesToPrune" AfterTargets="PrepforRestoreForComposeStore">
    <ItemGroup>
      <PackagesToPrune Include="Microsoft.CSharp" />
    </ItemGroup>
    <PropertyGroup>
      <PackagesToPrune>$(PackagesToPrune);@(PackagesToPrune)</PackagesToPrune>
    </PropertyGroup>
  </Target>
</Project>

Prune worked -- there are less packages in artifact.xml. Great.

Now, if I do this:

<Project Sdk="Microsoft.NET.Sdk.Web">
  <ItemGroup>
	<PackageReference Include="Microsoft.AspNetCore.Authentication.JwtBearer" Version="3.1.3" />
	<PackageReference Include="Amazon.Lambda.AspNetCoreServer" Version="5.0.0" />
  </ItemGroup>
  <Target Name="AddPackagesToPrune" AfterTargets="PrepforRestoreForComposeStore">
    <ItemGroup>
      <PackagesToPrune Include="Microsoft.CSharp" />
    </ItemGroup>
    <PropertyGroup>
      <PackagesToPrune>$(PackagesToPrune);@(PackagesToPrune)</PackagesToPrune>
    </PropertyGroup>
  </Target>
</Project>

I will get Microsoft.NET.ComposeStore.targets(420,5): error MSB4018: The "FilterResolvedFiles" task failed unexpectedly. error.

@davidruhmann
Copy link

@clearwaterstream Thanks for creating the proof of concept. I see, this error is because the Filter target is being run per dependency and the Amazon dependency is not pulling in Microsoft.CSharp. Have to step away, but wonder if we can inject in before the Filter target to dynamically build the prune list per dependency. Will take a look later but it will most likely hinge on how the FilterResolvedFiles target is written and if ResolvedFiles is accessible to a before target.
https://github.com/dotnet/sdk/blob/master/src/Tasks/Microsoft.NET.Build.Tasks/targets/Microsoft.NET.ComposeStore.targets#L420

@dsplaisted
Copy link
Member

We can probably fix the NullReferenceException in the 5.0 SDK if that would be a significant help. IE then it would be OK to specify packages to prune that aren't in the package graph.

@clearwaterstream
Copy link

@davidruhmann you mentioned "... dynamically build the prune list per dependency."

and @dsplaisted you mentioned "... specify packages to prune that aren't in the package graph".

So would the prune list calculation logic know how to not add packages to the prune list that are legitimate dependencies? For example:

<PackageReference Include="MySql.Data" Version="8.0.19" />

Would pull in (to name a few):

<Package Id="SSH.NET" Version="2016.1.0" />
<Package Id="SshNet.Security.Cryptography" Version="1.2.0" />
<Package Id="BouncyCastle.NetCore" Version="1.8.3" />
<Package Id="Google.Protobuf" Version="3.6.1" />

Currently if I blindly take all the "extra" packages that are in the resultant artifact.xml and add them to the prune list then I would be over-pruning in some cases.

@davidruhmann
Copy link

davidruhmann commented Apr 23, 2020 •
edited
Loading

@dsplaisted Yes, that would be nice.

@clearwaterstream To clarify, the list of excluded libraries for the use case is limited to what is found in the framework runtime (the BCLs). Sub-depedencies would not be part of the exclusion list unless manually specified.

Here is a working example excluding Microsoft.CSharp from artifact.xml; however the file still exists in the store. Instead of passing in PackagesToPrune, it is removing packages from the output ResolvedPackagesPublished.

<Project Sdk="Microsoft.NET.Sdk.Web">

  <ItemGroup>
    <PackageReference Include="Microsoft.AspNetCore.Authentication.JwtBearer" Version="3.1.3" />
    <PackageReference Include="Amazon.Lambda.AspNetCoreServer" Version="5.0.0" />
  </ItemGroup>

  <Target Name="PruneBCLs" AfterTargets="ComputeFilesToStore">
    <ItemGroup>
      <BCLsToPrune Include="Microsoft.CSharp" />
    </ItemGroup>
    <ItemGroup>
      <ResolvedPackagesPublished Remove="@(BCLsToPrune)" />
    </ItemGroup>
  </Target>

</Project>

Now, in the case that @normj originally posted above; the BCLsToPrune list could be populated with the filenames found in the runtime directory. (e.g. /usr/local/share/dotnet/shared/Microsoft.NETCore.App/3.1.3)

@normj
Copy link
Contributor Author

normj commented Apr 23, 2020

@dsplaisted Is there no way the NullReferenceException could be ported to the 3.1 SDK? We generally stick with LTS versions which means we wouldn't get a fix till .NET 6.

@dsplaisted dsplaisted mentioned this issue Apr 24, 2020
Merged
@dsplaisted
Copy link
Member

@normj The NullReferenceException should be fixed in the 3.1.400 SDK release.

@normj
Copy link
Contributor Author

normj commented May 11, 2020

@dsplaisted Great looking forward to it!

@niklr
Copy link

niklr commented May 23, 2020

I was in the process to give AWS Lambda layers with .NET Core a try and got stuck with the publish-layer command referencing this issue here. What are your current recommmendations for such a situation @normj ? Should we wait for 3.1.400 SDK to be released or should we try with TargetFramework=netcoreapp2.1 ? The latter seems not to work with the following dependencies:

  • Amazon.Lambda.APIGatewayEvents (v2.1.0)
  • Amazon.Lambda.Core (v1.1.0)
  • Amazon.Lambda.Serialization.SystemTextJs (v2.0.1)

@normj
Copy link
Contributor Author

normj commented May 26, 2020

@niklr For the Lambda side the ball is in my court now to update the Lambda tooling to use the work around added in 3.1.400. I hope to get to that soon just juggling a few things right now. I think the issue is still worth leaving open as I think the current fix in 3.1.400 is a work around that requires users of the dotnet store command to do extra work and hopefully there will be a fix someday that doesn't require any extra work.

@vkkpnm
Copy link

vkkpnm commented Jun 24, 2020

Looking forward for this change. Any tentative timelines?

@RNavaneethKumar
Copy link

@normj - Glad to see your contributions to the corresponding repositories this month. Can we expect anything soon?

@ppittle
Copy link

ppittle commented Jul 17, 2020

@dsplaisted - any news on when 3.1.400 will be out? Looks like the team released 3.1.302 earlier this week on July 14th 2020.

Verified version 3.1.302 still has the Null Reference Exception bug on @clearwaterstream's sample:

manifest.xml:

<Project Sdk="Microsoft.NET.Sdk">
  <ItemGroup>
    <PackageReference Include="Newtonsoft.Json" Version="9.0.1" />
	<PackageReference Include="Microsoft.AspNetCore.Authentication.JwtBearer" Version="3.1.3" />
	<PackageReference Include="Amazon.Lambda.AspNetCoreServer" Version="5.0.0" />
  </ItemGroup>
  <Target Name="AddPackagesToPrune" AfterTargets="PrepforRestoreForComposeStore">
    <ItemGroup>
      <PackagesToPrune Include="Microsoft.CSharp" />
    </ItemGroup>
    <PropertyGroup>
      <PackagesToPrune>$(PackagesToPrune);@(PackagesToPrune)</PackagesToPrune>
    </PropertyGroup>
  </Target>  
</Project>

console:

> dotnet --version
3.1.302
> dotnet store --manifest initial.xml -f netcoreapp3.1 -o bin/dotnetcore/store -r linux-x64 --skip-optimization

C:\Program Files\dotnet\sdk\3.1.302\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ComposeStore.targets(420,5): error MSB4018: The "FilterResolvedFiles" task failed unexpectedly. [C:\projects\sandbox\dotnetStoreBug\manifest.xml]
C:\Program Files\dotnet\sdk\3.1.302\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ComposeStore.targets(420,5): error MSB4018: System.NullReferenceException: Object reference not set to an instance of an object. [C:\projects\sandbox\dotnetStoreBug\manifest.xml]
C:\Program Files\dotnet\sdk\3.1.302\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ComposeStore.targets(420,5): error MSB4018:    at Microsoft.NET.Build.Tasks.LockFileExtensions.GetTransitivePackagesList(LockFileTarget lockFileTarget, LockFileTargetLibrary package, IDictionary`2 libraryLookup) [C:\projects\sandbox\dotnetStoreBug\manifest.xml]
C:\Program Files\dotnet\sdk\3.1.302\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ComposeStore.targets(420,5): error MSB4018:    at Microsoft.NET.Build.Tasks.ProjectContext.GetTransitiveList(String package) [C:\projects\sandbox\dotnetStoreBug\manifest.xml]
C:\Program Files\dotnet\sdk\3.1.302\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ComposeStore.targets(420,5): error MSB4018:    at Microsoft.NET.Build.Tasks.FilterResolvedFiles.ExecuteCore() [C:\projects\sandbox\dotnetStoreBug\manifest.xml]
C:\Program Files\dotnet\sdk\3.1.302\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ComposeStore.targets(420,5): error MSB4018:    at Microsoft.NET.Build.Tasks.TaskBase.Execute() [C:\projects\sandbox\dotnetStoreBug\manifest.xml]
C:\Program Files\dotnet\sdk\3.1.302\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ComposeStore.targets(420,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute() [C:\projects\sandbox\dotnetStoreBug\manifest.xml]
C:\Program Files\dotnet\sdk\3.1.302\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ComposeStore.targets(420,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask) [C:\projects\sandbox\dotnetStoreBug\manifest.xml]

@dsplaisted
Copy link
Member

@ppittle 3.1.400 will release together with Visual Studio 2019 version 16.7. I don't think we've announced when that will be, but you can see the previews we've released so far here: https://docs.microsoft.com/en-us/visualstudio/releases/2019/release-notes-preview

@clearwaterstream
Copy link

@dsplaisted I've skimmed through the notes https://docs.microsoft.com/en-us/visualstudio/releasenotes/vs2019-mac-preview-relnotes and 3.1.400 is not mentioned there either. Do you happen to know if this may potentially make it into the 8.7 release of VS 4 Mac?

@dsplaisted
Copy link
Member

@leecow Do you know if VS for Mac 8.7 will include .NET Core SDK 3.1.400? Thanks.

@leecow
Copy link
Member

leecow commented Jul 21, 2020

I believe that's correct. @mrward to confirm.

@mrward
Copy link
Contributor

mrward commented Jul 21, 2020

VS Mac 8.7 currently includes .NET 3.1.302.

I expect VS Mac 8.7 will at some point include .NET 3.1.400 since Visual Studio 2019 16.7 will be doing so.

@bjorg
Copy link

bjorg commented Jul 21, 2020

As far as I can tell, 3.1.400 is not released. If it is, I don't see it here: https://dotnet.microsoft.com/download/dotnet-core/3.1

@bonjo
Copy link

bonjo commented Jul 29, 2020 •
edited
Loading

@ppittle 3.1.400 will release together with Visual Studio 2019 version 16.7. I don't think we've announced when that will be, but you can see the previews we've released so far here: https://docs.microsoft.com/en-us/visualstudio/releases/2019/release-notes-preview

Hi @dsplaisted / @normj
Today I installed .NET Core 3.1.400-preview-015216 from here https://github.com/dotnet/installer#installers-and-binaries and I still getting the below message when I run the following command: dotnet lambda publish-layer CP-Layer --layer-type runtime-package-store --s3-bucket cp-site
Publishing runtime package store layers targeting .NET Core 3.1 is currently disabled due to an issue with the dotnet cli's 'store' command used to create the runtime package store.
Do you known something about when this issue will be solved?

@dsplaisted
Copy link
Member

@bonjo It sounds like that message is coming from the dotnet lambda tooling. IE it hasn't been updated to understand that the issue in the .NET SDK has been fixed.

@normj
Copy link
Contributor Author

normj commented Jul 29, 2020

@bonjo as @dsplaisted mentioned we (AWS) need to update the dotnet lambda tooling once 3.1.400 is released. Once the new version is released the work for updating the dotnet lambda tooling will be tracked here. aws/aws-extensions-for-dotnet-cli#113

@bonjo
Copy link

bonjo commented Jul 29, 2020

@normj I assume then that dotnet lambda tool doesn't have a preview version that works with .NET core 3.1.400 preview.
It's correct?

@bjorg
Copy link

bjorg commented Jul 29, 2020

@bonjo You're in luck! It's open source as well, so you can build it locally with the required modification to unblock yourself.
https://github.com/aws/aws-extensions-for-dotnet-cli

@normj
Copy link
Contributor Author

normj commented Jul 29, 2020

@bonjo correct we don't have a preview version as we haven't started the work yet.

@slavanap
Copy link

slavanap commented Aug 9, 2020

Whoever it may concern,
Please pay attention that with recent VS update MSI package for 3.1.400 is installed which is undeletable. Might be a bug with installation/uninstallation procedure. I don't know where to send this info.

@slavanap
Copy link

slavanap commented Aug 9, 2020

I would appreciate if someone provides official MSI for 3.1.400 (x64). The message below appears after MSVS uninstall.
Untitled

@dsplaisted
Copy link
Member

@slavanap We've released a new patch. You can download 3.1.401 here: https://dotnet.microsoft.com/download/dotnet-core/3.1

@devssharma
Copy link

I am getting this message while publishing the layer into AWS lambda layers using .NET CORE 3.1 Version.

image

Could you please help me how can i publish the layers into aws using .net core 3.1. ?? Do we have any workaround to do this ??

@bjorg
Copy link

bjorg commented Aug 19, 2020

@devssharma You need to post that over here: aws/aws-extensions-for-dotnet-cli#113
This issue was the underlying cause, but AWS needs to update their tool to re-enable it.

@normj normj mentioned this issue Oct 11, 2020
Merged
@normj
Copy link
Contributor Author

normj commented Oct 11, 2020

I have a PR out using the pruning workaround into the AWS Lambda tooling. aws/aws-extensions-for-dotnet-cli#144

@dsplaisted do you think there will ever be a better solution in the dotnet store command other then adding this pruning list? I hope so but if not then we can close this issue and people concern with the AWS use case should track the fix via the PR.

@dsplaisted
Copy link
Member

Unfortunately I think it's unlikely that we'll fix this with a better solution. It would require NuGet/Home#7344, which I would like to see and would fix some other issues. However, it would be a major feature for NuGet, and probably only helps for cases where a .NET Core 1.x / .NET Standard 1.x package is referenced, which should happen less and less as time goes on. So it's probably not going to be worth prioritizing.

@normj
Copy link
Contributor Author

normj commented Oct 12, 2020

Alright thanks for the pruning fix. That seems to work for our use case.

@normj normj closed this as completed Oct 12, 2020
@dsplaisted dsplaisted mentioned this issue Oct 6, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dsplaisted dsplaisted

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests