Classpath ordering in container dependent on host filesystem

[cpopp]

Environment:

• Jib version: 1.6.1
• Build tool: maven, 3.5.4
• OS: Ubuntu 18.04.5

Description of the issue:
The use of /app/libs/* in the classpath argument for java means that ordering of jars on the classpath is dependent on the host system. We had conflicting jars in our application, but it worked fine on our dev servers, however another set of servers had the jars ordered differently on the classpath and the jar conflict resulted in our application not starting.

Expected behavior:
We expect the containers to be as portable as possible between environments so we hoped the classpath would be ordered identically between environments which would occur if for example the jars were listed explicitly on the classpath rather than as /app/libs/*

Steps to reproduce:

1. Build image with Google jib
2. Run on one host and print the classpath
3. Run on another host with an alternative filesystem and print the classpath
4. See classpath order differs

jib-maven-plugin Configuration:

<plugin>
    <groupId>com.google.cloud.tools</groupId>
    <artifactId>jib-maven-plugin</artifactId>
    <version>1.6.1</version>
</plugin>

Additional Information:

Classpath wildcard iteration on a Linux host is shown here in the openjdk source: https://github.com/openjdk/jdk/blob/6bab0f539fba8fb441697846347597b4a0ade428/src/java.base/share/native/libjli/wildcard.c#L209

Which points to the use of readdir (https://www.man7.org/linux/man-pages/man3/readdir.3.html) which indicates:

The order in which filenames are read by successive calls to
readdir() depends on the filesystem implementation; it is unlikely
that the names will be sorted in any fashion.

which is exactly what we ran into when the option for mounting the host filesystem differed.

[chanseokoh]

Hi @cpopp,

This is a known issue: #1871 #1907 It's on our radar and something we really wanted to fix early, but we always had other priority issues.

The best is to avoid putting multiple different dependency JARs that happen to contain the same class of different versions on your classpath from the beginning. It is really risky to have the same class of different versions, as you observed in your prod env. So it's ideal to fix your dependencies; you never know when those classes will be loaded in a different order in the future on a different environment. If fixing that is not possible, sometimes you could tell Maven to exclude one JAR like this example. Other workarounds could be to set your own <entrypoint>, use a wrapper script, or create a classpath input parameter file if you are on recent Java versions.

Lastly, please use 2.5.2 instead of 1.6.1.

Classpath wildcard iteration on a Linux host is shown here in the openjdk source: https://github.com/openjdk/jdk/blob/6bab0f539fba8fb441697846347597b4a0ade428/src/java.base/share/native/libjli/wildcard.c#L209

Which points to the use of readdir (https://www.man7.org/linux/man-pages/man3/readdir.3.html) which indicates:

The order in which filenames are read by successive calls to
readdir() depends on the filesystem implementation; it is unlikely
that the names will be sorted in any fashion.

which is exactly what we ran into when the option for mounting the host filesystem differed.

Thanks for the info. Personally, I've been always curious of the exact behavior (of OpenJDK at least).

Closing as a dup of #1871. Please follow up there.

[cpopp]

Reading the other issues they seemed to use alternative ways of running their application and then noticed the issue when attempting to use jib. In this issue the varying classpath orderings both came from jib runs using the exact same image (just on different host systems). While the core fix may be the same, I think considering the portability of a Docker image makes this a bit different.

[loosebazooka]

@saturnism has suggested that you never should have duplicate classes in your classpath. Use something like https://www.mojohaus.org/extra-enforcer-rules/banDuplicateClasses.html to enforce this on your projects.

[chanseokoh]

In any case, we will fix #1871, probably by explicitly listing dependency JARs. Hopefully, we may not have to worry too much about the classpath string length.

I have lost the confidence.

[cpopp]

@loosebazooka in our case the issue was multiple slf4j logging implementations (log4j and logback) that conflicted and not actually duplicate classes. It dynamically chose the implementation based on which it encountered first, and the application bailed with an error when one was used but worked fine with the other. And yes, of course we should not have both on the classpath and have since fixed it since we encountered this issue messing up a planned deploy to the next environment.

[saturnism]

@cpopp slf4j impl are also troublesome :( it'd be good to exclude the duplicate impl based on the warning msg.

#1871 will still be useful though, by listing out individual JARs in the classpath will actually help w/ #2471

[chanseokoh]

@saturnism @cpopp we've released Jib 2.7.0 which added a new configuration option (jib.container.expandClasspathDependencies (Gradle) / <container><expandClasspathDependencies> (Maven)) that enables expanding classpath dependencies in the default java command for an image ENTRYPOINT. Turning on the option (off by default) will enumerate all the dependencies, which will match the dependency loading order in Maven or Gradle builds. For example, the ENTRYPOINT becomes

java ... -cp /app/resources:/app/classes:/app/libs/spring-boot-starter-web-2.0.3.RELEASE.jar:/app/libs/shared-library-0.1.0.jar:/app/libs/spring-boot-starter-json-2.0.3.RELEASE.jar:... com.example.Main

instead of the default

java ... -cp /app/resources:/app/classes:/app/libs/* com.example.Main

Expanding the dependency list can be useful in AppCDS too.

Note that an expanded dependency list can become very long in practice, and we are not sure if there may be a potential issue due to a long command line ("argument list too long" or "command line is too long").

As with other Jib configurations, this option can also be set through the system property (-Djib.container.expandClasspathDependencies=true|false).

[chanseokoh]

@saturnism @cpopp Jib 3.1.1 is released, which creates two JVM argument files inside an image. One of them is the Java runtime classpath where all the dependencies are explicitly enumerated, which enables Jib to preseve the depending loading order by using this file for Java 9+. (For Java 8, you have to continue to set expandClasspathDependencies to true.)

For those interested, see here for more details.

[andreagalle]

Just a simple questio, was this issue somehow related to jib itself, or you opened this isssue here just because you reproduced it using jib? Shouldn't this behaviour be independent from the docker image build process itself?

[chanseokoh]

@andreagalle the issue was set, with Java 8 or below, Jib sets java ... --classpath ...:/app/libs/*:... for the container image entrypiont. And the way a JVM expands * depends on host platforms.