Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make Extensions linker friendly. #40431

Merged
merged 5 commits into from
Aug 7, 2020
Merged

Conversation

eerhardt
Copy link
Member

@eerhardt eerhardt commented Aug 5, 2020

Annotating the rest of the Microsoft.Extensions libraries.

Fix #40396

With these changes, there are the following linker warnings left in Microsoft.Extensions assemblies:

https://github.com/mono/linker/issues/1412
F:\git\runtime2\src\libraries\Microsoft.Extensions.Configuration.Abstractions\src\ConfigurationExtensions.cs(22,9): Trim analysis warning IL2006: Microsoft.Extensions.Configuration.ConfigurationExtensions.Add<TSource>(IConfigurationBuilder,Action<TSource>): The generic parameter 'TSource' from 'Microsoft.Extensions.Configuration.ConfigurationExtensions.Add<TSource>(IConfigurationBuilder,Action<TSource>)' with dynamically accessed member kinds 'None' is passed into the generic parameter 'T' from 'System.Activator.CreateInstance<T>()' which requires dynamically accessed member kinds 'PublicParameterlessConstructor'. To fix this add DynamicallyAccessedMembersAttribute to it and specify at least these member kinds 'PublicParameterlessConstructor'.

https://github.com/mono/linker/issues/1416
F:\git\runtime2\src\libraries\Microsoft.Extensions.Http\src\DependencyInjection\HttpClientBuilderExtensions.cs(339,13): Trim analysis warning IL2006: Microsoft.Extensions.DependencyInjection.HttpClientBuilderExtensions.<>c__DisplayClass10_0<TClient>.<AddTypedClientCore>b__0(IServiceProvider): The generic parameter 'TClient' from 'Microsoft.Extensions.DependencyInjection.HttpClientBuilderExtensions.<>c__DisplayClass10_0<TClient>' with dynamically accessed member kinds 'None' is passed into the generic parameter 'TClient' from 'Microsoft.Extensions.Http.ITypedHttpClientFactory<TClient>' which requires dynamically accessed member kinds 'PublicConstructors'. To fix this add DynamicallyAccessedMembersAttribute to it and specify at least these member kinds 'PublicConstructors'.
F:\git\runtime2\src\libraries\Microsoft.Extensions.Http\src\DependencyInjection\HttpClientBuilderExtensions.cs(339,13): Trim analysis warning IL2006: Microsoft.Extensions.DependencyInjection.HttpClientBuilderExtensions.<>c__DisplayClass10_0<TClient>.<AddTypedClientCore>b__0(IServiceProvider): The generic parameter 'TClient' from 'Microsoft.Extensions.DependencyInjection.HttpClientBuilderExtensions.<>c__DisplayClass10_0<TClient>' with dynamically accessed member kinds 'None' is passed into the generic parameter 'TClient' from 'Microsoft.Extensions.Http.ITypedHttpClientFactory<TClient>' which requires dynamically accessed member kinds 'PublicConstructors'. To fix this add DynamicallyAccessedMembersAttribute to it and specify at least these member kinds 'PublicConstructors'.
F:\git\runtime2\src\libraries\Microsoft.Extensions.Http\src\DependencyInjection\HttpClientBuilderExtensions.cs(339,13): Trim analysis warning IL2006: Microsoft.Extensions.DependencyInjection.HttpClientBuilderExtensions.<>c__DisplayClass10_0<TClient>.<AddTypedClientCore>b__0(IServiceProvider): The generic parameter 'TClient' from 'Microsoft.Extensions.DependencyInjection.HttpClientBuilderExtensions.<>c__DisplayClass10_0<TClient>' with dynamically accessed member kinds 'None' is passed into the generic parameter 'TClient' from 'Microsoft.Extensions.Http.ITypedHttpClientFactory<TClient>' which requires dynamically accessed member kinds 'PublicConstructors'. To fix this add DynamicallyAccessedMembersAttribute to it and specify at least these member kinds 'PublicConstructors'.
F:\git\runtime2\src\libraries\Microsoft.Extensions.Http\src\DependencyInjection\HttpClientBuilderExtensions.cs(402,13): Trim analysis warning IL2006: Microsoft.Extensions.DependencyInjection.HttpClientBuilderExtensions.<>c__DisplayClass12_0<TClient,TImplementation>.<AddTypedClientCore>b__0(IServiceProvider): The generic parameter 'TImplementation' from 'Microsoft.Extensions.DependencyInjection.HttpClientBuilderExtensions.<>c__DisplayClass12_0<TClient,TImplementation>' with dynamically accessed member kinds 'None' is passed into the generic parameter 'TClient' from 'Microsoft.Extensions.Http.ITypedHttpClientFactory<TClient>' which requires dynamically accessed member kinds 'PublicConstructors'. To fix this add DynamicallyAccessedMembersAttribute to it and specify at least these member kinds 'PublicConstructors'.
F:\git\runtime2\src\libraries\Microsoft.Extensions.Http\src\DependencyInjection\HttpClientBuilderExtensions.cs(402,13): Trim analysis warning IL2006: Microsoft.Extensions.DependencyInjection.HttpClientBuilderExtensions.<>c__DisplayClass12_0<TClient,TImplementation>.<AddTypedClientCore>b__0(IServiceProvider): The generic parameter 'TImplementation' from 'Microsoft.Extensions.DependencyInjection.HttpClientBuilderExtensions.<>c__DisplayClass12_0<TClient,TImplementation>' with dynamically accessed member kinds 'None' is passed into the generic parameter 'TClient' from 'Microsoft.Extensions.Http.ITypedHttpClientFactory<TClient>' which requires dynamically accessed member kinds 'PublicConstructors'. To fix this add DynamicallyAccessedMembersAttribute to it and specify at least these member kinds 'PublicConstructors'.
F:\git\runtime2\src\libraries\Microsoft.Extensions.Http\src\DependencyInjection\HttpClientBuilderExtensions.cs(402,13): Trim analysis warning IL2006: Microsoft.Extensions.DependencyInjection.HttpClientBuilderExtensions.<>c__DisplayClass12_0<TClient,TImplementation>.<AddTypedClientCore>b__0(IServiceProvider): The generic parameter 'TImplementation' from 'Microsoft.Extensions.DependencyInjection.HttpClientBuilderExtensions.<>c__DisplayClass12_0<TClient,TImplementation>' with dynamically accessed member kinds 'None' is passed into the generic parameter 'TClient' from 'Microsoft.Extensions.Http.ITypedHttpClientFactory<TClient>' which requires dynamically accessed member kinds 'PublicConstructors'. To fix this add DynamicallyAccessedMembersAttribute to it and specify at least these member kinds 'PublicConstructors'.

ConfigurationBinder recursively inspects properties - this can't be modeled with the linker - https://github.com/mono/linker/issues/1087
F:\git\runtime2\src\libraries\Microsoft.Extensions.Configuration.Binder\src\ConfigurationBinder.cs(371,17): Trim analysis warning IL2006: Microsoft.Extensions.Configuration.ConfigurationBinder.CreateInstance(Type): The parameter 'type' of method 'Microsoft.Extensions.Configuration.ConfigurationBinder.CreateInstance(Type)' with dynamically accessed member kinds 'None' is passed into the implicit 'this' parameter of method 'System.Reflection.TypeInfo.get_DeclaredConstructors()' which requires dynamically accessed member kinds 'NonPublicConstructors | PublicConstructors'. To fix this add DynamicallyAccessedMembersAttribute to it and specify at least these member kinds 'NonPublicConstructors | PublicConstructors'.
F:\git\runtime2\src\libraries\Microsoft.Extensions.Configuration.Binder\src\ConfigurationBinder.cs(380,17): Trim analysis warning IL2006: Microsoft.Extensions.Configuration.ConfigurationBinder.CreateInstance(Type): The parameter 'type' of method 'Microsoft.Extensions.Configuration.ConfigurationBinder.CreateInstance(Type)' with dynamically accessed member kinds 'None' is passed into the parameter #0 of method 'System.Object System.Activator::CreateInstance(System.Type)' which requires dynamically accessed member kinds 'PublicParameterlessConstructor'. To fix this add DynamicallyAccessedMembersAttribute to it and specify at least these member kinds 'PublicParameterlessConstructor'.
F:\git\runtime2\src\libraries\Microsoft.Extensions.Configuration.Binder\src\ConfigurationBinder.cs(403,13): Trim analysis warning IL2006: Microsoft.Extensions.Configuration.ConfigurationBinder.BindDictionary(Object,Type,IConfiguration,BinderOptions): The parameter 'dictionaryType' of method 'Microsoft.Extensions.Configuration.ConfigurationBinder.BindDictionary(Object,Type,IConfiguration,BinderOptions)' with dynamically accessed member kinds 'None' is passed into the implicit 'this' parameter of method 'System.Reflection.TypeInfo.GetDeclaredProperty(String)' which requires dynamically accessed member kinds 'NonPublicProperties | PublicProperties'. To fix this add DynamicallyAccessedMembersAttribute to it and specify at least these member kinds 'NonPublicProperties | PublicProperties'.
F:\git\runtime2\src\libraries\Microsoft.Extensions.Configuration.Binder\src\ConfigurationBinder.cs(433,13): Trim analysis warning IL2006: Microsoft.Extensions.Configuration.ConfigurationBinder.BindCollection(Object,Type,IConfiguration,BinderOptions): The parameter 'collectionType' of method 'Microsoft.Extensions.Configuration.ConfigurationBinder.BindCollection(Object,Type,IConfiguration,BinderOptions)' with dynamically accessed member kinds 'None' is passed into the implicit 'this' parameter of method 'System.Reflection.TypeInfo.GetDeclaredMethod(String)' which requires dynamically accessed member kinds 'NonPublicMethods | PublicMethods'. To fix this add DynamicallyAccessedMembersAttribute to it and specify at least these member kinds 'NonPublicMethods | PublicMethods'.
F:\git\runtime2\src\libraries\Microsoft.Extensions.Configuration.Binder\src\ConfigurationBinder.cs(565,17): Trim analysis warning IL2006: Microsoft.Extensions.Configuration.ConfigurationBinder.GetAllProperties(TypeInfo): The parameter 'type' of method 'Microsoft.Extensions.Configuration.ConfigurationBinder.GetAllProperties(TypeInfo)' with dynamically accessed member kinds 'None' is passed into the implicit 'this' parameter of method 'System.Reflection.TypeInfo.get_DeclaredProperties()' which requires dynamically accessed member kinds 'NonPublicProperties | PublicProperties'. To fix this add DynamicallyAccessedMembersAttribute to it and specify at least these member kinds 'NonPublicProperties | PublicProperties'.

The following are all 'MakeGenericMethod' - I'm not sure what we can do about these
F:\git\runtime2\src\libraries\Microsoft.Extensions.DependencyInjection\src\ServiceLookup\CallSiteFactory.cs(241,17): Trim analysis warning IL2006: Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteFactory.TryCreateOpenGeneric(ServiceDescriptor,Type,CallSiteChain,Int32): Calling to 'System.Type.MakeGenericType' on unrecognized value.
F:\git\runtime2\src\libraries\Microsoft.Extensions.DependencyInjection\src\ServiceLookup\Expressions\ExpressionResolverBuilder.cs(135,17): Trim analysis warning IL2006: Microsoft.Extensions.DependencyInjection.ServiceLookup.ExpressionResolverBuilder.VisitIEnumerable(IEnumerableCallSite,Object): Call to 'System.Reflection.MethodInfo.MakeGenericMethod' is not recognized.
F:\git\runtime2\src\libraries\Microsoft.Extensions.DependencyInjection\src\ServiceLookup\ILEmit\ILEmitResolverBuilder.cs(216,17): Trim analysis warning IL2006: Microsoft.Extensions.DependencyInjection.ServiceLookup.ILEmitResolverBuilder.VisitIEnumerable(IEnumerableCallSite,ILEmitResolverBuilderContext): Call to 'System.Reflection.MethodInfo.MakeGenericMethod' is not recognized.

@davidfowl
Copy link
Member

The following are all 'MakeGenericMethod' - I'm not sure what we can do about these

I think we have to suppress these. There's nothing we can flow from here. Any open generic registered into the container needs to do the needful and mark things appropriately.

@davidfowl
Copy link
Member

ConfigurationBinder recursively inspects properties - this can't be modeled with the linker - dotnet/linker#1087

This is unfortunate but understandable (especially because it's polymorphic)

@MichalStrehovsky
Copy link
Member

For the warnings that refer to the DisplayClass - we could work around by adding our own closure class that is properly annotated. Linker can't do much without trying to reverse engineer C# compiler operations and that's a sketchy territory (I don't think the Roslyn team would be happy if we were to take a dependency on their manglings).

For MakeGenericMethod - the risk is basically that if we can end up with MakeGenericMethod creating generic methods that have generic parameters annotated with dataflow annotations, linker no longer enforces the annotations (because linker doesn't run at runtime) and things could start failing. If we're convinced that can't happen, suppression is okay.

Basically suppression always means that things could start failing at runtime because we're turning off automatic checking. It needs to be done with extreme care because one missed spot and everything could fall apart with a hard to diagnose exception at runtime. We also need to remember that a method got MakeGenericMethod'd somewhere and we should not add dataflow annotations to it in the future because they won't be enforced.

@davidfowl
Copy link
Member

For the warnings that refer to the DisplayClass - we could work around by adding our own closure class that is properly annotated. Linker can't do much without trying to reverse engineer C# compiler operations and that's a sketchy territory (I don't think the Roslyn team would be happy if we were to take a dependency on their manglings).

Why can't we do something automagical for compiler generated types (like inherit attributes when they are from compiler generated types).

For MakeGenericMethod - the risk is basically that if we can end up with MakeGenericMethod creating generic methods that have generic parameters annotated with dataflow annotations, linker no longer enforces the annotations (because linker doesn't run at runtime) and things could start failing. If we're convinced that can't happen, suppression is okay.

Right, the way this system works it make sense to suppress. It can break but we'll need to document this pattern in general (open generics registered in DI have this problem and there's no single fix).

@MichalStrehovsky
Copy link
Member

Why can't we do something automagical for compiler generated types (like inherit attributes when they are from compiler generated types).

It requires threading the annotations through to the closure constructor, and to the instance fields. It's not impossible but it requires careful implementation of the right constructs that the C# compiler can emit for these.

We've been avoiding doing crossmethod dataflow analysis because it's slow to compute and starts to be complicated to both compute and report warnings on once things like virtual and interface methods are supported. Closure classes might be less complicated, but it still requires hardcoding C# compiler knowledge I would rather not hardcode.

@davidfowl
Copy link
Member

It needs to be solved though, if we change people's coding patterns because they can't write annotations on closures, it basically means you'll get warnings that need to be ignored forever. Forcing me to rewrite my code to add an annotation is also not great.

@MichalStrehovsky
Copy link
Member

The code needs to be written in a way that can be statically analyzed. This pull request is doing it in e.g. ProviderAliasUtilities.cs already.

There will be reflection code that will work as-is. There will be code where you add annotations and it will work. There will be code that you need to tweak before you can annotate it. There will be code that you need to throw out and write a source generator.

C# has a lot of these closure-based constructs where it will generate a lot of goo that can only be analyzed with whole program dataflow for something that is a one-liner in the C# source. There's yield return state machines. There's async state machines. There's lambda closures. There's the dynamic keyword. It's a lot of fragile concepts to hardcode into linker and C# is not done adding more.

@MichalStrehovsky
Copy link
Member

ConfigurationBinder recursively inspects properties

What's the failure mode when the properties are missing?

I have bad memories of missing properties in XAML databinding in .NET Native - XAML is pretty quiet when a property doesn't exist so a bug where we trimmed something that shouldn't have been trimmed usually manifested itself with "the UI looks off, but there's no exception thrown anywhere". Those were fun to debug.

@davidfowl
Copy link
Member

C# has a lot of these closure-based constructs where it will generate a lot of goo that can only be analyzed with whole program dataflow for something that is a one-liner in the C# source. There's yield return state machines. There's async state machines. There's lambda closures. There's the dynamic keyword. It's a lot of fragile concepts to hardcode into linker and C# is not done adding more.

Asking people rewrite their code is a non starter. It would be better if there were more well defined contracts defined between the linker and the compiler (that's why I initially focused on CompilerGeneratedAttribute).

@eerhardt eerhardt requested a review from layomia August 7, 2020 14:59
@eerhardt eerhardt merged commit 254ef0f into dotnet:master Aug 7, 2020
@eerhardt eerhardt deleted the AnnotateExtensions branch August 7, 2020 19:36
@eerhardt
Copy link
Member Author

eerhardt commented Aug 7, 2020

ConfigurationBinder recursively inspects properties

What's the failure mode when the properties are missing?

From what I can tell it silently skips the properties without setters. So you can start getting NullReferenceException if you were expecting the properties to be set to valid objects after binding.

I've opened #40551 to make the Binder linker-safe.

It would be better if there were more well defined contracts defined between the linker and the compiler

Would it be possible for the compiler to flow the generic type attributes to the generated code? That doesn't seem unreasonable to me, and I think it could solve the problem with HttpClientBuilderExtensions.

@jkoritzinsky
Copy link
Member

Would it be possible for the compiler to flow the generic type attributes to the generated code? That doesn't seem unreasonable to me, and I think it could solve the problem with HttpClientBuilderExtensions.

That should be feasible and seems like a clean solution. Let's open an issue with the Roslyn team about it.

@eerhardt
Copy link
Member Author

eerhardt commented Aug 7, 2020

Let's open an issue with the Roslyn team about it.

I've opened dotnet/roslyn#46646 for this.

@vitek-karas
Copy link
Member

@eerhardt Why didn't we also update the ref assemblies with the new attributes?

Jacksondr5 pushed a commit to Jacksondr5/runtime that referenced this pull request Aug 10, 2020
* Make Extensions linker friendly.

Annotating the rest of the Microsoft.Extensions libraries.

Fix dotnet#40396

* Disable parallelism in trimming tests to fix race condition.
@eerhardt
Copy link
Member Author

Why didn't we also update the ref assemblies with the new attributes?

Mostly because I forgot to. I've opened #40621 which is adding them.

As an aside - these assemblies don't target net5.0, so the attributes in the ref assembly are all internal. Also, we have an infrastructure bug that isn't catching these problems on attributes on parameters and on generic types. cc @safern

@karelz karelz modified the milestones: 6.0.0, 5.0.0 Aug 18, 2020
@ghost ghost locked as resolved and limited conversation to collaborators Dec 7, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Make Microsoft.Extensions.Hosting linker friendly
8 participants