Unified handling of Resolve calls #1979

marek-safar · 2021-04-20T09:45:34Z

This change introduces a commonplace for resolving definition for types,
methods and fields. This is useful for following reasons

Speeds up linker by 20% for default Blazor app (more for large apps)
Any custom step can avoid building local mapping cache
Custom steps could support --skip-unresolved linker option
Consistent error handling for unresolved references

Most of the changes are just mechanical method replacement and
Link context passing.

Contributes to #918
Fixes #1953

This change introduces a commonplace for resolving definition for types, methods and fields. This is useful for following reasons - Speeds up linker by 20% for default Blazor app (more for large apps) - Any custom step can avoid building local mapping cache - Custom steps could support `--skip-unresolved` linker option - Consistent error handling for unresolved references Most of the changes are just mechanical method replacement and Link context passing. Contributes to dotnet#918 Fixes dotnet#1953

src/linker/Linker.Dataflow/ValueNode.cs

src/linker/Linker.Dataflow/ReflectionMethodBodyScanner.cs

src/linker/Linker/LinkContext.cs

vitek-karas · 2021-04-20T10:08:34Z

src/linker/Linker/LinkContext.cs

+			if (typeReference is TypeSpecification ts) {
+				if (typeReference is FunctionPointerType) {
+					td = null;
+				} else {
+					//
+					// It returns element-type for arrays and also element type for wrapping types like ByReference, PinnedType, etc
+					//
+					td = TryResolveTypeDefinition (ts.GetElementType ());
+				}


Should we create an issue to handle at least some of these at the callsite - resolving to element-type is dangerous. Not counting that it also hides generic instantiations.

I actually experimented with it a bit and I think this logic is fine. Most code still needs to resolve to the element so the separation would not have benefits beyond perhaps type system purity.

src/linker/Linker/LinkContext.cs

src/linker/Linker.Steps/MarkStep.cs

Co-authored-by: Vitek Karas <[email protected]>

src/linker/Linker.Dataflow/ValueNode.cs

This just gets dotnet/linker#1979 out of the way since it's very noisy and doesn't apply to us.

@jonpryor

This will allow extension methods that take an `IMetadataResolver` like in dotnet/java-interop#842 to use the LinkContext Resolve cache. Context: dotnet/android#5748 (comment) The Resolve cache added in dotnet#1979 requires calling `Resolve*Definition` methods directly on `LinkContext`, which means that any extension methods that do resolution logic need to take a `LinkContext`. This doesn't work well with the layering in xamarin-android, where java.interop uses a resolution cache with cecil, but doesn't depend on the linker. Instead it uses a custom `TypeResolutionCache` for extension methods like `GetBaseDefinition`: https://github.com/xamarin/java.interop/blob/main/src/Java.Interop.Tools.Cecil/Java.Interop.Tools.Cecil/MethodDefinitionRocks.cs#L16 These extension methods are also used from xamarin-android, but there's a desire to use the `LinkContext` cache in this case. @jonpryor had the idea to change the extension methods to use cecil's `IMetadataResolver`, which can be implemented by `TypeDefinitionCache` and by `LinkContext`. Java.interop will continue using their `TypeDefinitionCache`, and xamarin-android will use `LinkContext`. One limitation of this approach is that `LinkContext.TryResolve*Definition` (renamed to just `TryResolve` for consistency) methods aren't usable from the extension methods.

@jonpryor

This will allow extension methods that take an `IMetadataResolver` like in dotnet/java-interop#842 to use the LinkContext Resolve cache. Context: dotnet/android#5748 (comment) The Resolve cache added in #1979 requires calling `Resolve*Definition` methods directly on `LinkContext`, which means that any extension methods that do resolution logic need to take a `LinkContext`. This doesn't work well with the layering in xamarin-android, where java.interop uses a resolution cache with cecil, but doesn't depend on the linker. Instead it uses a custom `TypeResolutionCache` for extension methods like `GetBaseDefinition`: https://github.com/xamarin/java.interop/blob/main/src/Java.Interop.Tools.Cecil/Java.Interop.Tools.Cecil/MethodDefinitionRocks.cs#L16 These extension methods are also used from xamarin-android, but there's a desire to use the `LinkContext` cache in this case. @jonpryor had the idea to change the extension methods to use cecil's `IMetadataResolver`, which can be implemented by `TypeDefinitionCache` and by `LinkContext`. Java.interop will continue using their `TypeDefinitionCache`, and xamarin-android will use `LinkContext`. One limitation of this approach is that `LinkContext.TryResolve*Definition` (renamed to just `TryResolve` for consistency) methods aren't usable from the extension methods.

Context: dotnet/linker#1953 Context: dotnet/linker#1774 Context: dotnet/linker#1774 (comment) Context: dotnet/linker#1979 Context: dotnet/linker#2019 Context: dotnet/linker#2045 Context: dotnet/java-interop@2573dc8 Context: #5870 Context: #5878 Update the custom linker steps to use the new `IMarkHandler` API which runs logic during "MarkStep". (See [this list][0] of pipeline steps for additional context.) As part of this, I've removed the workaround that loads all referenced assemblies up-front and simplified some of the linker MSBuild targets. Some of the `MonoDroid.Tuner` steps were duplicated and changed to implement the new `IMarkHandler` interface, to avoid touching the `MonoDroid.Tuner` code. In my analysis of the custom steps, most of them "just work" if we call them only for marked members, because they ultimately either: - Just call `AddPreserved*()` to conditionally keep members of types (which with the new API will just happen when the type is marked) - In the case of `FixAbstractMethodsStep()`, inject missing interface implementations which will only be kept if they are marked for some other reason. Most of the steps have been updated in a straightforward way based on the above. The exceptions are: - `AddKeepAlivesStep` needs to run on all code that survived linking, and can run as a normal step. - `ApplyPreserveAttribute`: this step globally marks members with `PreserveAttribute`. - The step is only active for non-SDK link assemblies. Usually we root non-SDK assemblies, but it will be a problem if linking all assemblies. - For now, I updated it to use the new custom step API on assembly mark -- so it will scan for the attribute in all marked assemblies -- but we should investigate whether this is good enough. - This can't be migrated to use `IMarkHandler` because it needs to scan code for attributes, including unmarked code. - `PreserveExportedTypes`: similar to the above, this globally marks members with `Export*Attribute`. It's used for java interop in cases where the java methods aren't referenced statically, like from xml, or for special serialization methods. - It's only active for non-SDK assemblies, so like above it will be a problem only if linking all assemblies. - Like above, I've made it scan marked assemblies. - `PreserveApplications`: globally scans for `ApplicationAttribute` on types/assemblies, and sets the `TypePreserve` annotation for any types referenced by the attribute. - This step technically does a global scan, but it's likely to work on "mark type"/"mark assembly", as `ApplicationAttribute` is only used on types/assembies that are already kept. - I've updated it to use the new `IMarkHandler` API. Additionally, as per dotnet/java-interop@2573dc8c, stop using `TypeDefinitionCache` everywhere and instead use `IMetadataResolver` to support caching `TypeDefinition`s across shared steps. For .NET 6, `LinkContextMetadataResolver` implements the `IMetadataResolver` interface in terms of `LinkContext`. [0]: https://github.com/mono/linker/blob/main/src/linker/Linker/Driver.cs#L714-L730

* Unified handling of Resolve calls This change introduces a commonplace for resolving definition for types, methods and fields. This is useful for following reasons - Speeds up linker by 20% for default Blazor app (more for large apps) - Any custom step can avoid building local mapping cache - Custom steps could support `--skip-unresolved` linker option - Consistent error handling for unresolved references Most of the changes are just mechanical method replacement and Link context passing. Contributes to dotnet/linker#918 Fixes dotnet/linker#1953 * Update src/linker/Linker/LinkContext.cs Co-authored-by: Vitek Karas <[email protected]> * PR feedback * Simplify MethodReturnValue node construction Co-authored-by: Vitek Karas <[email protected]> Commit migrated from dotnet/linker@012efef

@jonpryor

This will allow extension methods that take an `IMetadataResolver` like in dotnet/java-interop#842 to use the LinkContext Resolve cache. Context: dotnet/android#5748 (comment) The Resolve cache added in dotnet/linker#1979 requires calling `Resolve*Definition` methods directly on `LinkContext`, which means that any extension methods that do resolution logic need to take a `LinkContext`. This doesn't work well with the layering in xamarin-android, where java.interop uses a resolution cache with cecil, but doesn't depend on the linker. Instead it uses a custom `TypeResolutionCache` for extension methods like `GetBaseDefinition`: https://github.com/xamarin/java.interop/blob/main/src/Java.Interop.Tools.Cecil/Java.Interop.Tools.Cecil/MethodDefinitionRocks.cs#L16 These extension methods are also used from xamarin-android, but there's a desire to use the `LinkContext` cache in this case. @jonpryor had the idea to change the extension methods to use cecil's `IMetadataResolver`, which can be implemented by `TypeDefinitionCache` and by `LinkContext`. Java.interop will continue using their `TypeDefinitionCache`, and xamarin-android will use `LinkContext`. One limitation of this approach is that `LinkContext.TryResolve*Definition` (renamed to just `TryResolve` for consistency) methods aren't usable from the extension methods. Commit migrated from dotnet/linker@aaf4880

marek-safar requested a review from vitek-karas April 20, 2021 09:45

vitek-karas approved these changes Apr 20, 2021

View reviewed changes

marek-safar and others added 2 commits April 20, 2021 13:31

Update src/linker/Linker/LinkContext.cs

51000c5

Co-authored-by: Vitek Karas <[email protected]>

PR feedback

4b044db

vitek-karas reviewed Apr 20, 2021

View reviewed changes

src/linker/Linker.Dataflow/ValueNode.cs Outdated Show resolved Hide resolved

Simplify MethodReturnValue node construction

ae67472

vitek-karas approved these changes Apr 20, 2021

View reviewed changes

marek-safar merged commit 012efef into dotnet:main Apr 22, 2021

marek-safar deleted the resolve branch April 22, 2021 07:26

This was referenced Apr 27, 2021

Update to new linker custom steps API dotnet/android#5748

Merged

Update to new linker custom steps API xamarin/xamarin-macios#11374

Merged

MichalStrehovsky added a commit to MichalStrehovsky/runtimelab that referenced this pull request Apr 29, 2021

Sync files shared with IL Linker

1b9def0

This just gets dotnet/linker#1979 out of the way since it's very noisy and doesn't apply to us.

MichalStrehovsky mentioned this pull request Apr 29, 2021

Sync files shared with IL Linker dotnet/runtimelab#1036

Merged

jkotas pushed a commit to dotnet/runtimelab that referenced this pull request Apr 29, 2021

Sync files shared with IL Linker (#1036)

28ddf89

This just gets dotnet/linker#1979 out of the way since it's very noisy and doesn't apply to us.

sbomer mentioned this pull request May 5, 2021

[.NET 6] Linker steps should use built-in resolve cache xamarin/xamarin-macios#11448

Open

sbomer mentioned this pull request May 19, 2021

Implement IMetadataResolver for LinkContext #2045

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unified handling of Resolve calls #1979

Unified handling of Resolve calls #1979

marek-safar commented Apr 20, 2021

vitek-karas Apr 20, 2021

marek-safar Apr 20, 2021

Unified handling of Resolve calls #1979

Unified handling of Resolve calls #1979

Conversation

marek-safar commented Apr 20, 2021

vitek-karas Apr 20, 2021

Choose a reason for hiding this comment

marek-safar Apr 20, 2021

Choose a reason for hiding this comment