Can't load jffi because of "/lib64/libc.so.6: version `GLIBC_2.27' not found"

[rosti-il]

After the docker-maven-plugin was updated from version 0.40.2 into version 0.42.0 it fails to run during Jenkins pipelines of my employer. The error message looks like in the following example:

[2023-03-20T09:28:08.635Z] [ERROR] -----------------------------------------------------: could not get native definition for type `POINTER`, original error message follows: java.lang.UnsatisfiedLinkError: Unable to execute or load jffi binary stub from `/tmp`. Set `TMPDIR` or Java property `java.io.tmpdir` to a read/write path that is not mounted "noexec".
[2023-03-20T09:28:08.635Z] [ERROR] /tmp/jenkins/workspace/xxx/yyy/zzz/MR-Open/service/jffi8656036966694745987.so: /lib64/libc.so.6: version `GLIBC_2.27' not found (required by /tmp/jenkins/workspace/xxx/yyy/zzz/MR-Open/service/jffi8656036966694745987.so)
[2023-03-20T09:28:08.635Z] [ERROR] 	at com.kenai.jffi.internal.StubLoader.tempLoadError(StubLoader.java:555)
[2023-03-20T09:28:08.635Z] [ERROR] 	at com.kenai.jffi.internal.StubLoader.loadFromJar(StubLoader.java:454)
[2023-03-20T09:28:08.635Z] [ERROR] 	at com.kenai.jffi.internal.StubLoader.load(StubLoader.java:330)
[2023-03-20T09:28:08.636Z] [ERROR] 	at com.kenai.jffi.internal.StubLoader.<clinit>(StubLoader.java:618)

The reason why it happens is that some of the libjffi-1.2.so library files packaged within the jffi-1.3.10-native.jar transitive dependency Jar file was linked with GLIBC version 2.27, so they fail to load on Linux/UNIX systems with an older version of GLIBC.

This jffi-1.3.10-native.jar transitive dependency (com.github.jnr:jffi:jar:native:1.3.10) comes from the com.github.jnr:jnr-unixsocket:jar:0.38.19 that is the latest now

[INFO] +- com.github.jnr:jnr-unixsocket:jar:0.38.19:compile
[INFO] |  +- com.github.jnr:jnr-ffi:jar:2.2.13:compile
[INFO] |  |  +- com.github.jnr:jffi:jar:1.3.10:compile
[INFO] |  |  +- com.github.jnr:jffi:jar:native:1.3.10:runtime

However there is a newer version of the com.github.jnr:jffi:jar:native - version 1.3.11. One of the changes of this version is following commit: jnr/jffi@821cbf0 with following commit message:

Try using debian:8 for older glibc

Building on debian:10 results in a binary that requires glibc 2.27
which is not available on some LTS server Linuxes like CentOS 7.
This commit attempts to make a compatible jffi.so using debian:8,
which ships with glibc 2.19. This is not as old as 2.17 but may
remove enough dependencies on newer glibc to be sufficient.

See #138

This a not completed fix of the jnr/jffi#138 bug.

Why it's not completed? Because it fixes the issue for the x86_64-Linux platform only. Comparison of the jffi-1.3.10-native.jar and the newer jffi-1.3.11-native.jar shows differences in x86_64-Linux and x86_64-OpenBSD platforms only.

The OpenBSD OS isn't related to this issue but there are many other Linux platforms with the libjffi-1.2.so library still linked with GLIBC_2.27 and Jenkins of my employer seems using one of them and not x86_64-Linux.

To examine a libjffi-1.2.so file and find what version of GLIBC it was linked with in Linux you can use its ldd tool. If you use Windows a quick and dirty way to check it is just looking for the GLIBC_ strings inside the binary content of the libjffi-1.2.so file. For example:

/jni/x86_64-Linux/libjffi-1.2.so in jffi-1.3.10-native.jar contains GLIBC_2.27:

/jni/x86_64-Linux/libjffi-1.2.so in jffi-1.3.11-native.jar doesn't contain GLIBC_2.27:

/jni/aarch64-Linux/libjffi-1.2.so in jffi-1.3.11-native.jar still contains GLIBC_2.27:

And the same bugfix incompleteness in several other Linux platforms.

If you use x86_64-Linux the following workaround is reported as working:

            <plugin>
                <groupId>io.fabric8</groupId>
                <artifactId>docker-maven-plugin</artifactId>
                <version>0.42.0</version>
                <dependencies>
                    <dependency>
                        <groupId>com.github.jnr</groupId>
                        <artifactId>jffi</artifactId>
                        <version>1.3.11</version>
                    </dependency>
                    <dependency>
                        <groupId>com.github.jnr</groupId>
                        <artifactId>jffi</artifactId>
                        <version>1.3.11</version>
                        <classifier>native</classifier>
                    </dependency>
                </dependencies>
            </plugin>

But in other Linux platforms it doesn't because of the above incompleteness of bug fix in the jffi project.

[rhuss]

@rosti-il thanks a lot for this indepth investigation, that is very helpful !

@rohanKanojia I propose that we make a new release of dmp, to include at least the fix for linux-amd64 (I guess its not so much an issue on Darwin anyways, as I suppose to have newer glibc deps anyway)

The alternative would be to downgrade jiffi. @rohanKanojia do you remember for what fix/feature we have upgraded Jiffi in dmp ?

[rohanKanojia]

I think it seems to be coming from com.github.jnr:jnr-unixsocket . I had bumped this dependency in #1638

[rohanKanojia]

@rosti-il : I've released 0.42.1 with updated jffi dependencies. Could you please check whether this fixes issue on your Jenkins?

[visit1985]

that fixed it for me on CentOS 7

[rohanKanojia]

@visit1985 : Thanks a lot for your feedback!

[DaPutzy]

that fixed it for me on CentOS 7

Can confirm!

[rosti-il]

@rosti-il : I've released 0.42.1 with updated jffi dependencies. Could you please check whether this fixes issue on your Jenkins?

Yes, it fixed the issue. Thank you!