AIX libjnidispatch is wrong format #1066

Taywee · 2019-02-13T00:45:16Z

Provide complete information about the problem

Version of JNA and related jars

jna and jna-platform are 5.2.0

Version and vendor of the java virtual machine

OpenJDK 1.8.0_202 with HotspotVM

Operating system

AIX 7.1 64-bit

System architecture (CPU type, bitness of the JVM)

PowerPC 64

Complete description of the problem

When I try to use JNA on AIX 64-bit (though it will probably work as well the same way in 32-bit, judging by the library format), I get this exception (my code paths truncated at the end):

Exception in thread "main" java.lang.UnsatisfiedLinkError: Native library (com/sun/jna/aix-ppc64/libjnidispatch.so) not found in resource path ([file:/opt/sysshepjr/agent/sysshepjr.jar])
        at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:1026)
        at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:988)
        at com.sun.jna.Native.<clinit>(Native.java:195)
        at com.sun.jna.platform.unix.LibC.<clinit>(LibC.java:35)

Looking at the library stated, it is in there in .a format. Like many Linux/Unix systems, AIX allows linking against .so shared objects and .a static libraries, but it also allows .a archives to store .so shared objects to allow "smart switching" based on bitness (allowing one archive to have both the 32-bit and 64-bit libraries). The library that is in the release is named .a, but it is not an ar archive, it is a shared object renamed to .a (including the 32-bit version). There are two obvious options for fixing this:

Rename the .a back to .so for both 32-bit and 64-bit, allowing the loader to correctly find it. This is preferable, because JNA already distinguishes based on path the bit size.
Rename the .a back to .so for both, and pack them in a .a archive, and correctly refer to the archive by the AIX contained member style. This makes the library more in the style of typical AIX libraries, but doesn't really bring any advantages to the table.

matthiasblaesing · 2019-02-13T06:07:21Z

This is an issue tracker for confirmed issues - I doubt that there is really a problem in JNA. As mentioned in the issue template, the google group would be the appropriate place to ask.

Judging from the class path (only a single jar) I assume, that you repacked JNA, make sure, that the aix binaries are packed along (or don't do it!):

        0  2018-12-23 12:18   com/sun/jna/aix-ppc/
   487425  2018-07-15 11:55   com/sun/jna/aix-ppc/libjnidispatch.a
        0  2018-12-23 12:18   com/sun/jna/aix-ppc64/
   526244  2018-07-15 11:35   com/sun/jna/aix-ppc64/libjnidispatch.a

Please rerun with the system property jna.debug_load.jna set to true:
https://github.com/java-native-access/jna/blob/master/www/FrequentlyAskedQuestions.md#calling-nativeload-causes-an-unsatisfiedlinkerror

Taywee · 2019-02-13T16:21:34Z

I verified that the files were in there. As I said, they are in there and in the wrong format. .a is an archive format on AIX, either containing object files (as a static library) or shared objects (to be a shared library), never as an executable object file:

$ unzip jna-5.2.0.jar com/sun/jna/aix-ppc64/libjnidispatch.a com/sun/jna/aix-ppc/libjnidispatch.a 
Archive:  jna-5.2.0.jar
  inflating: com/sun/jna/aix-ppc/libjnidispatch.a  
  inflating: com/sun/jna/aix-ppc64/libjnidispatch.a  

$ file com/sun/jna/aix-ppc/libjnidispatch.a com/sun/jna/aix-ppc64/libjnidispatch.a
com/sun/jna/aix-ppc/libjnidispatch.a:   executable (RISC System/6000 V3.1) or obj module not stripped
com/sun/jna/aix-ppc64/libjnidispatch.a: 64-bit XCOFF executable or object module

This is the wrong format for a .a file for both architectures. Renaming them to .so and repacking them fixes it, as would switching them to a correct archive format and perhaps changing the load path. This is definitely an issue in the released jar.

edit: Also, sorry if this is the wrong place. I can re-post it somewhere else if it is supposed to be verified by somebody else before coming here.

matthiasblaesing · 2019-02-13T17:28:41Z

The AIX libraries are packaged that way for a long time and were verified to work (I also verified myself on 32bit and 64bit):
#964

Please run:
wget https://repo1.maven.org/maven2/net/java/dev/jna/jna/5.2.0/jna-5.2.0.jar
java -jar jna-5.2.0.jar

And post the output here. I currently don't have access to an AIX machine, so can't test myself.

Taywee · 2019-02-13T19:09:02Z

$ /opt/openjdk/java/bin/java -Djna.debug_load.jna=true -jar ./jna-5.2.0.jar
Feb 13, 2019 1:14:11 PM com.sun.jna.Native extractFromResourcePath
INFO: Looking in classpath from sun.misc.Launcher$AppClassLoader@70dea4e for /com/sun/jna/aix-ppc64/libjnidispatch.so
Exception in thread "main" java.lang.UnsatisfiedLinkError: Native library (com/sun/jna/aix-ppc64/libjnidispatch.so) not found in resource path ([file:/home/tcr/jna-5.2.0.jar])
        at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:1026)
        at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:988)
        at com.sun.jna.Native.<clinit>(Native.java:195)

It does appear to work with openj9, though (which is the default JVM on AIX, previously being IBM J9):

$ /usr/java8_64/bin/java -jar jna-5.2.0.jar
Java Native Access (JNA) API Version 5
Version: 5.2.0 (b0)
 Native: 6.0.0 (147a998f0cbc89681a1ae6c0dd121629)
 Prefix: aix-ppc64

Actually, if I change out the .a for .so, it works for OpenJDK and fails for OpenJ9, so this actually has different behavior on different JVMs:

$ unzip jna-5.2.0.jar com/sun/jna/aix-ppc64/libjnidispatch.a com/sun/jna/aix-ppc/libjnidispatch.a
Archive:  jna-5.2.0.jar
  inflating: com/sun/jna/aix-ppc/libjnidispatch.a  
  inflating: com/sun/jna/aix-ppc64/libjnidispatch.a  

$ mv com/sun/jna/aix-ppc64/libjnidispatch.a com/sun/jna/aix-ppc64/libjnidispatch.so

$ mv com/sun/jna/aix-ppc/libjnidispatch.a com/sun/jna/aix-ppc/libjnidispatch.so

$ zip -d jna-5.2.0.jar com/sun/jna/aix-ppc64/libjnidispatch.a com/sun/jna/aix-ppc/libjnidispatch.a 
deleting: com/sun/jna/aix-ppc/libjnidispatch.a
deleting: com/sun/jna/aix-ppc64/libjnidispatch.a

$ zip -u jna-5.2.0.jar com/sun/jna/aix-ppc64/libjnidispatch.so com/sun/jna/aix-ppc/libjnidispatch.so
  adding: com/sun/jna/aix-ppc64/libjnidispatch.so (deflated 75%)
  adding: com/sun/jna/aix-ppc/libjnidispatch.so (deflated 73%)

$ /opt/openjdk/java/bin/java -jar ./jna-5.2.0.jar
Java Native Access (JNA) API Version 5
Version: 5.2.0 (b0)
 Native: 6.0.0 (147a998f0cbc89681a1ae6c0dd121629)
 Prefix: aix-ppc64

$ /usr/java8_64/bin/java -jar ./jna-5.2.0.jar 
Exception in thread "main" java.lang.UnsatisfiedLinkError: Native library (com/sun/jna/aix-ppc64/libjnidispatch.a) not found in resource path ([file:/home/tcr/jna-5.2.0.jar])
        at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:1026)
        at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:988)
        at com.sun.jna.Native.<clinit>(Native.java:195)

Either way, a shared object named .a is definitely incorrect on AIX, whether it incidentally works or not.

matthiasblaesing · 2019-02-13T19:26:47Z

Thanks for the investigation. I don't know when I'll get access to an AIX machine again and even then I doubt, that I'll be able to work with OpenJDK on that, so I won't be able to help much here. Given, that it works with the platforms JDK, I see an issue in OpenJDK (actually to my memory OpenJ9 is OpenJDK + the IBM VM).

My reading is a bit different than yours: Shared libraries on AIX can be embedded in a archive (".a") file. The ".a" files embed shared libraries.

Taywee · 2019-02-13T20:12:57Z

Yes, that's absolutely true, but these .a files are not archives. They are .so files renamed to .a.

$ file /usr/lib/libssl.a
/usr/lib/libssl.a: archive (big format)

$ ar t /usr/lib/libssl.a
libssl.so
libssl.so.0.9.8
libssl.so.1.0.0

$ unzip jna-5.2.0.jar com/sun/jna/aix-ppc64/libjnidispatch.a com/sun/jna/aix-ppc/libjnidispatch.a
Archive:  jna-5.2.0.jar
  inflating: com/sun/jna/aix-ppc/libjnidispatch.a  
  inflating: com/sun/jna/aix-ppc64/libjnidispatch.a  

$ file com/sun/jna/aix-ppc/libjnidispatch.a com/sun/jna/aix-ppc64/libjnidispatch.a
com/sun/jna/aix-ppc/libjnidispatch.a: executable (RISC System/6000) or object module not stripped
com/sun/jna/aix-ppc64/libjnidispatch.a: 64-bit XCOFF executable or object module not stripped

$ ar t com/sun/jna/aix-ppc64/libjnidispatch.a
ar: 0707-108 File com/sun/jna/aix-ppc64/libjnidispatch.a is not an archive file.

.a files are always archives an AIX, whether they contain a shared library or static objects. These files in the jna distribution are not archive files.

It seems System#mapLibraryName from OpenJDK works differently on AIX than OpenJ9. OpenJ9 expands jnidispatch to libjnidispatch.a, while OpenJDK expands it to libjnidispatch.so. This in turn causes the loader to search for the wrong file. java-native-access#1066

matthiasblaesing · 2019-02-14T22:15:35Z

After more careful reading, the problem is not the library loader, but the library name mapper. For OpenJ9 and older JDKs on AIX System.mapLibraryName("jnidispatch") obvisually returns the correct libjnidispatch.a, while OpenJDK returns libjnidispatch.so

I added a manual override here:

matthiasblaesing@40c099a

Can you build from source and test?

In any case, it would be interesting to run

public class MappingCheck {
    public static void main(String[] args) {
	System.out.println(System.mapLibraryName("jnidispatch"));
    }
}

If OpenJ9 and OpenJDK really return different values, this should be reported as bug against OpenJDK, as that is the new kid on the platform.

Taywee · 2019-02-14T22:48:15Z

It's also an issue with the library name. These libraries are named .a, but they are not archive files. .a files are never executable objects in AIX, but these ones are.

matthiasblaesing · 2019-02-14T22:56:44Z

For years this worked correctly. I won't throw away working code that fails on a corner case (openjdk). Please either/and a) provide a working patch b) provide documentation that supports your claim, that the solution working for years is wrong c) provide testing for suggested fix/further info Greetings Matthias Am 14. Februar 2019 23:48:18 MEZ schrieb "Taylor C. Richberger" <[email protected]>:

…

It's also an issue with the library name. These libraries are named .a, but they are not archive files. .a files are never executable objects in AIX, but these ones are. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: #1066 (comment)

-- Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.

Taywee · 2019-02-14T23:00:31Z

It looks like they do return different library names between different JVMs:

$ /usr/java8_64/bin/java MappingCheck
libjnidispatch.a

$ /opt/openjdk/java/bin/java MappingCheck
libjnidispatch.so

Either can be correct, though, as it just "Maps a library name into a platform-specific string representing a native library.", and a native shared library can be either format for AIX, either as a shared object or an archive containing shared objects.

Documentation: https://www.ibm.com/support/knowledgecenter/SSGH2K_13.1.0/com.ibm.xlc131.aix.doc/proguide/compiling_shared_aix.html

The .a format it shows is called an "archive library file", made using the ar command.

matthiasblaesing · 2019-02-14T23:03:32Z

So we established, that OpenJDK differs from OpenJ9. Are you able to test my patch? Am 15. Februar 2019 00:00:36 MEZ schrieb "Taylor C. Richberger" <[email protected]>:

…

It looks like they do return different library names between different JVMs: ```sh $ /usr/java8_64/bin/java MappingCheck libjnidispatch.a $ /opt/openjdk/java/bin/java MappingCheck libjnidispatch.so ``` Either can be correct, though, as it just "Maps a library name into a platform-specific string representing a native library.", and a native shared library can be either format for AIX, either as a shared object or an archive containing shared objects. Documentation: https://www.ibm.com/support/knowledgecenter/SSGH2K_13.1.0/com.ibm.xlc131.aix.doc/proguide/compiling_shared_aix.html The .a format it shows is called an "archive library file", made using the `ar` command. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: #1066 (comment)

-- Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.

Taywee · 2019-02-14T23:22:08Z

So after checking with an IBM guy, it turns out that I was at least partially wrong. .so is always a shared object type, and .a is usually an archive of object files for static linking or shared objects, but may also just be a shared object on its own. My mistake there. I apologize. I'll see if I can check that patch real fast.

Taywee · 2019-02-14T23:36:41Z

That patch looks like it works just fine:

bash-4.4$ /usr/java8_64/bin/java -jar ./jna.jar       
Java Native Access (JNA) API Version 5
Version: 5.2.1-SNAPSHOT (b0)
 Native: 6.0.0 (147a998f0cbc89681a1ae6c0dd121629)
 Prefix: aix-ppc64

bash-4.4$ /opt/openjdk/java/bin/java -jar ./jna.jar 
Java Native Access (JNA) API Version 5
Version: 5.2.1-SNAPSHOT (b0)
 Native: 6.0.0 (147a998f0cbc89681a1ae6c0dd121629)
 Prefix: aix-ppc64
bash-4.4$

Again, sorry for the frustration.

matthiasblaesing · 2019-02-15T20:43:47Z

Thank you for investigating and testing! I'll update the patch with a change log entry and merge it to master.

It seems System#mapLibraryName from OpenJDK works differently on AIX than OpenJ9. OpenJ9 expands jnidispatch to libjnidispatch.a, while OpenJDK expands it to libjnidispatch.so. This in turn causes the loader to search for the wrong file. java-native-access#1066 Closes: java-native-access#1066

It seems System#mapLibraryName from OpenJDK works differently on AIX than J9. J9 expands jnidispatch to libjnidispatch.a, while OpenJDK expands it to libjnidispatch.so. This in turn causes the loader to search for the wrong file. java-native-access#1066 Closes: java-native-access#1066

matthiasblaesing mentioned this issue Feb 15, 2019

Use correct native library on OpenJDK on AIX #1067

Merged

matthiasblaesing closed this as completed in #1067 Feb 16, 2019

mawinter69 mentioned this issue Jun 6, 2019

[JENKINS-57515] update JNA library from 4.5.2 to 5.3.1 jenkinsci/jenkins#4060

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AIX libjnidispatch is wrong format #1066

AIX libjnidispatch is wrong format #1066

Taywee commented Feb 13, 2019

matthiasblaesing commented Feb 13, 2019

Taywee commented Feb 13, 2019 •

edited

Loading

matthiasblaesing commented Feb 13, 2019

Taywee commented Feb 13, 2019 •

edited

Loading

matthiasblaesing commented Feb 13, 2019

Taywee commented Feb 13, 2019 •

edited

Loading

matthiasblaesing commented Feb 14, 2019

Taywee commented Feb 14, 2019

matthiasblaesing commented Feb 14, 2019 via email

Taywee commented Feb 14, 2019

matthiasblaesing commented Feb 14, 2019 via email

Taywee commented Feb 14, 2019 •

edited

Loading

Taywee commented Feb 14, 2019

matthiasblaesing commented Feb 15, 2019

AIX libjnidispatch is wrong format #1066

AIX libjnidispatch is wrong format #1066

Comments

Taywee commented Feb 13, 2019

Provide complete information about the problem

matthiasblaesing commented Feb 13, 2019

Taywee commented Feb 13, 2019 • edited Loading

matthiasblaesing commented Feb 13, 2019

Taywee commented Feb 13, 2019 • edited Loading

matthiasblaesing commented Feb 13, 2019

Taywee commented Feb 13, 2019 • edited Loading

matthiasblaesing commented Feb 14, 2019

Taywee commented Feb 14, 2019

matthiasblaesing commented Feb 14, 2019 via email

Taywee commented Feb 14, 2019

matthiasblaesing commented Feb 14, 2019 via email

Taywee commented Feb 14, 2019 • edited Loading

Taywee commented Feb 14, 2019

matthiasblaesing commented Feb 15, 2019

Taywee commented Feb 13, 2019 •

edited

Loading

Taywee commented Feb 13, 2019 •

edited

Loading

Taywee commented Feb 13, 2019 •

edited

Loading

Taywee commented Feb 14, 2019 •

edited

Loading