[GR-48129] [GR-48285] Method handle and foreign function fixes and documentation.

docs/reference-manual/native-image/ForeignInterface.md

@@ -5,31 +5,34 @@ link_title: Foreign Interface
 permalink: /reference-manual/native-image/dynamic-features/foreign-interface/
 ---
 
-# Foreign Interface in Native Image
+# Foreign Function & Memory API in Native Image
 
-The Foreign Interface is a native API that enables Java code to interact with native code and vice versa.
-It is currently a preview API of the Java platform and must be enabled with `--enable-preview`.
-This page gives an overview of its support in Native Image.
+The Foreign Function & Memory (FFM) API is a native interface that enables Java code to interact with native code and vice versa.
+As of [JEP 442](https://openjdk.org/jeps/442){:target="_blank"}, it is a preview API of the Java platform and must be enabled with `--enable-preview`.
+Modules that are permitted to perform "restricted" native operations (including creating handles for calls to or from native code) must be specified using `--enable-native-access=`.
+This page gives an overview of support for the FFM API in Native Image.
 
 ## Foreign memory
-Shared arenas are not supported.
+Foreign memory functionality is generally supported. Shared arenas are currently not supported.
 
 ## Foreign functions
-The Foreign Functions Interface (FFI) allows Java code to call native functions, and conversely allows native code to invoke Java method handles.
-These two kind of calls are referred to as "downcalls" and "upcalls" respectively and are collectively referred to as "foreign calls".
+The FFM API enables Java code to call _down_ to native functions, and conversely allows native code to call _up_ to invoke Java code via method handles.
+These two kinds of calls are referred to as "downcalls" and "upcalls" respectively and are collectively referred to as "foreign calls".
 
-This feature is currently only supported on the AMD64 platform.
+Currently, only downcalls are supported, and only on the AMD64 architecture.
 
 ### Looking up native functions
-FFI provides the `SymbolLookup` interface which allows to search native libraries for functions by name.
-`loaderLookup` is currently the only supported `SymbolLookup`.
+The FFM API provides the `SymbolLookup` interface to find functions in native libraries by name.
+`SymbolLookup.loaderLookup()` is currently the only supported kind of `SymbolLookup`.
 
 ### Registering foreign calls
-In order to perform a call to native, some glue code is required and thus must be generated at build time.
-Therefore, a list of the types of downcall which will be performed must be provided to the `native-image` builder.
+In order to perform calls to native code at runtime, supporting code must be generated at image build time.
+Therefore, the `native-image` tool must be provided with descriptors that characterize functions to which downcalls may be performed at runtime.
 
-This list can be specified using a custom `Feature`. For example:
+These descriptors can be registered using a custom `Feature`, for example:
 ```java
+import static java.lang.foreign.ValueLayout.*;
+
 class ForeignRegistrationFeature implements Feature { 
   public void duringSetup(DuringSetupAccess access) {
     RuntimeForeignAccess.registerForDowncall(FunctionDescriptor.ofVoid());
@@ -40,7 +43,9 @@ class ForeignRegistrationFeature implements Feature {
   }
 }
 ```
-To activate the custom feature `--features=<fully qualified name of ForeignRegistrationFeature class>` needs to be passed to native-image.
-[Native Image Build Configuration](BuildConfiguration.md#embed-a-configuration-file) explains how this can be automated with a `native-image.properties` file in `META-INF/native-image`.
+To activate the custom feature, `--features=com.example.ForeignRegistrationFeature` (the fully-qualified name of the feature class) needs to be passed to `native-image`.
+It is recommended to do so [with a _native-image.properties_ file](BuildConfiguration.md#embed-a-configuration-file).
+
+### Upcalls
 
-Upcalls are currently not supported.
+Upcalls are not yet supported.

substratevm/CHANGELOG.md

@@ -13,13 +13,14 @@ This changelog summarizes major changes to GraalVM Native Image.
 * (GR-38994) Together with Red Hat, we added support for `-XX:+HeapDumpOnOutOfMemoryError`.
 * (GR-47365) Throw `MissingReflectionRegistrationError` when attempting to create a proxy class without having it registered at build-time, instead of a `VMError`.
 * (GR-46064) Add option `-H:±IndirectBranchTargetMarker` to mark indirect branch targets on AMD64 with an endbranch instruction. This is a prerequisite for future Intel CET support.
-* (GR-46740) Add support for foreign downcalls (part of "Project Panama") on the AMD64 platform.
+* (GR-46740) Preview of [Foreign Function & Memory API downcalls](https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/ForeignInterface.md) (part of "Project Panama", [JEP 442](https://openjdk.org/jeps/442)) on AMD64. Must be enabled with `--enable-preview`.
 * (GR-27034) Add `-H:ImageBuildID` option to generate Image Build ID, which is a 128-bit UUID string generated randomly, once per bundle or digest of input args when bundles are not used.
 * (GR-47647) Add `-H:±UnlockExperimentalVMOptions` for unlocking access to experimental options similar to HotSpot's `-XX:UnlockExperimentalVMOptions`. Explicit unlocking will be required in a future release, which can be tested with the env setting `NATIVE_IMAGE_EXPERIMENTAL_OPTIONS_ARE_FATAL=true`. For more details, see [issue #7105](https://github.com/oracle/graal/issues/7105).
 * (GR-47647) Add `--color[=WHEN]` option to color the output WHEN ('always', 'never', or 'auto'). This API option supersedes the experimental option `-H:+BuildOutputColorful`.
 * (GR-43920) Add support for executing native image bundles as jar files with extra options `--with-native-image-agent` and `--container`.
 * (GR-43920) Add `,container[=<container-tool>]`, `,dockerfile=<dockerfile>` and `,dry-run` options to `--bundle-create`and `--bundle-apply`.
 * (GR-46420) Switch to directly using cgroup support from the JDK.
+* (GR-29688) More sophisticated intrinsification and inlining of method handle usages, both explicit and implicit (lambdas, string concatenation and record classes), and various fixes for non-intrinsified usages.
 
 ## GraalVM for JDK 17 and GraalVM for JDK 20 (Internal Version 23.0.0)
 * (GR-40187) Report invalid use of SVM specific classes on image class- or module-path as error. As a temporary workaround, `-H:+AllowDeprecatedBuilderClassesOnImageClasspath` allows turning the error into a warning.

substratevm/src/com.oracle.svm.hosted/src/com/oracle/svm/hosted/methodhandles/MethodHandleFeature.java

@@ -42,6 +42,7 @@
 
 import com.oracle.graal.pointsto.meta.AnalysisMetaAccess;
 import com.oracle.graal.pointsto.meta.AnalysisType;
+import com.oracle.svm.core.SubstrateOptions;
 import com.oracle.svm.core.feature.AutomaticallyRegisteredFeature;
 import com.oracle.svm.core.feature.InternalFeature;
 import com.oracle.svm.core.fieldvaluetransformer.FieldValueTransformerWithAvailability;
@@ -140,9 +141,15 @@ public void duringSetup(DuringSetupAccess access) {
             referencedKeySetAdd = ReflectionUtil.lookupMethod(concurrentWeakInternSetClass, "add", Object.class);
         }
 
-        var accessImpl = (DuringSetupAccessImpl) access;
-        substitutionProcessor = new MethodHandleInvokerRenamingSubstitutionProcessor(accessImpl.getBigBang());
-        accessImpl.registerSubstitutionProcessor(substitutionProcessor);
+        if (!SubstrateOptions.UseOldMethodHandleIntrinsics.getValue()) {
+            /*
+             * Renaming is not crucial with old method handle intrinsics, so if those are requested
+             * explicitly, disable renaming to offer a fallback in case it causes problems.
+             */
+            var accessImpl = (DuringSetupAccessImpl) access;
+            substitutionProcessor = new MethodHandleInvokerRenamingSubstitutionProcessor(accessImpl.getBigBang());
+            accessImpl.registerSubstitutionProcessor(substitutionProcessor);
+        }
     }
 
     @Override

substratevm/src/com.oracle.svm.hosted/src/com/oracle/svm/hosted/methodhandles/MethodHandleInvokerRenamingSubstitutionProcessor.java

@@ -24,6 +24,7 @@
  */
 package com.oracle.svm.hosted.methodhandles;
 
+import java.lang.reflect.Field;
 import java.lang.reflect.Method;
 import java.util.HashSet;
 import java.util.List;
@@ -47,8 +48,11 @@
  * the {@code LambdaForm} which they were compiled from.
  */
 public class MethodHandleInvokerRenamingSubstitutionProcessor extends SubstitutionProcessor {
-    private static final Class<?> LAMBDA_FORM_CLASS = ReflectionUtil.lookupClass(false, "java.lang.invoke.LambdaForm");
     private static final Method CLASS_GET_CLASS_DATA_METHOD = ReflectionUtil.lookupMethod(Class.class, "getClassData");
+    private static final Class<?> LAMBDA_FORM_CLASS = ReflectionUtil.lookupClass(false, "java.lang.invoke.LambdaForm");
+    private static final Field LAMBDA_FORM_CUSTOMIZED_FIELD = ReflectionUtil.lookupField(LAMBDA_FORM_CLASS, "customized");
+    private static final Class<?> DIRECT_METHOD_HANDLE_CLASS = ReflectionUtil.lookupClass(false, "java.lang.invoke.DirectMethodHandle");
+    private static final Method DIRECT_METHOD_HANDLE_INTERNAL_MEMBER_NAME_METHOD = ReflectionUtil.lookupMethod(DIRECT_METHOD_HANDLE_CLASS, "internalMemberName");
 
     /*
      * We currently only replace the invokers of direct method handles which have a simpler
@@ -88,15 +92,39 @@ public ResolvedJavaType resolve(ResolvedJavaType type) {
 
     private ResolvedJavaType getSubstitution(ResolvedJavaType type) {
         return typeSubstitutions.computeIfAbsent(type, original -> {
+            Object lambdaForm;
+            Object customizedMemberName = null;
             try {
                 Class<?> clazz = OriginalClassProvider.getJavaClass(original);
                 Object classData = CLASS_GET_CLASS_DATA_METHOD.invoke(clazz);
-                VMError.guarantee(LAMBDA_FORM_CLASS.isInstance(classData));
-                int hash = classData.hashCode();
-                return new MethodHandleInvokerSubstitutionType(original, findUniqueName(hash));
+                if (LAMBDA_FORM_CLASS.isInstance(classData)) {
+                    lambdaForm = classData;
+                } else if (classData instanceof List<?> list && list.size() == 2) {
+                    lambdaForm = list.get(0);
+                    Object customizedHandle = list.get(1);
+                    VMError.guarantee(LAMBDA_FORM_CLASS.isInstance(lambdaForm) && DIRECT_METHOD_HANDLE_CLASS.isInstance(customizedHandle) &&
+                                    LAMBDA_FORM_CUSTOMIZED_FIELD.get(lambdaForm) == customizedHandle, "Expected classData to contain LambdaForm and its customization: %s", classData);
+                    customizedMemberName = DIRECT_METHOD_HANDLE_INTERNAL_MEMBER_NAME_METHOD.invoke(customizedHandle);
+                } else {
+                    throw VMError.shouldNotReachHere("Unexpected classData: %s", classData);
+                }
             } catch (ReflectiveOperationException e) {
                 throw VMError.shouldNotReachHere(e);
             }
+            /*
+             * LambdaForm.hashCode() is not stable between image builds because it incorporates
+             * identity hash codes of objects such as those of Class<?> that don't override
+             * hashCode(). For that reason, we compute a hash code from LambdaForm.toString(). It
+             * might also not be perfectly unique because the string contains unqualified class
+             * names and can contain string representations of constraints that may be arbitrary
+             * objects, but it should typically be distinct and stable.
+             */
+            int hash = lambdaForm.toString().hashCode();
+            if (customizedMemberName != null) {
+                /* MemberName.hashCode() also includes identity hash codes of Class<?> objects. */
+                hash = hash * 31 + customizedMemberName.toString().hashCode();
+            }
+            return new MethodHandleInvokerSubstitutionType(original, findUniqueName(hash));
         });
     }