Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Class Loader still used while it's supposed to be closed #41233

Open
gsmet opened this issue Jun 16, 2024 · 9 comments
Open

Class Loader still used while it's supposed to be closed #41233

gsmet opened this issue Jun 16, 2024 · 9 comments

Comments

@gsmet
Copy link
Member

gsmet commented Jun 16, 2024

As part of my work trying to improve the CL situation, I stumbled upon a few cases of $TITLE for which I would need some help.

In my patch, I introduce a safe guard throwing an exception when you are trying to load a class from a CL that has been previously closed.
The rationale is that we shouldn't load any class from a CL that has been closed.

I already fixed a few of these problems... but I still have some going on that are a lot less easy to pinpoint.

These issues where all observed in this specific CI job run: https://github.com/gsmet/quarkus/actions/runs/9535653448/job/26281846636

  1. io.vertx.core.impl.VertxImpl$1.operationComplete
2024-06-16 12:05:59,044 WARN [io.net.uti.con.DefaultPromise] (globalEventExecutor-1-1) An exception was thrown by io.vertx.core.impl.VertxImpl$1.operationComplete(): java.lang.IllegalStateException: Class loader Augmentation Class Loader: PROD for minikube-with-defaults has been closed and may not be accessed anymore
at io.quarkus.bootstrap.classloading.QuarkusClassLoader.ensureOpen(QuarkusClassLoader.java:723)
at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:499)
at io.vertx.core.impl.VertxImpl$1.operationComplete(VertxImpl.java:986)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625)
at io.netty.util.concurrent.DefaultPromise.setSuccess(DefaultPromise.java:97)
at io.netty.util.concurrent.MultithreadEventExecutorGroup$1.operationComplete(MultithreadEventExecutorGroup.java:117)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557)
at io.netty.util.concurrent.DefaultPromise.access$200(DefaultPromise.java:35)
at io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:503)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
at io.netty.util.concurrent.GlobalEventExecutor$TaskRunner.run(GlobalEventExecutor.java:262)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:1583)

This one is due to the delay to handle the graceful shutdown:

From io.vertx.core.impl.VertxImpl:

       eventLoopGroup.shutdownGracefully(0, 10, TimeUnit.SECONDS).addListener(new GenericFutureListener() {
            @Override
            public void operationComplete(io.netty.util.concurrent.Future future) throws Exception {
              if (!future.isSuccess()) {
                log.warn("Failure in shutting down event loop group", future.cause());
              }
              if (metrics != null) {
                metrics.close();
              }
              if (tracer != null) {
                tracer.close();
              }

              checker.close();

              if (completionHandler != null) {
                eventLoopThreadFactory.newThread(() -> {
                  completionHandler.handle(Future.succeededFuture());
                }).start();
              }
            }
          });

A more complete stacktrace:

[INFO] java.lang.IllegalArgumentException: io.vertx.core.impl
[INFO]     at java.lang.ClassLoader.definePackage (ClassLoader.java:2215)
[INFO]     at io.quarkus.bootstrap.classloading.QuarkusClassLoader.definePackage (QuarkusClassLoader.java:576)
[INFO]     at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass (QuarkusClassLoader.java:542)
[INFO]     at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass (QuarkusClassLoader.java:501)
[INFO]     at java.lang.ClassLoader.defineClass1 (Native Method)
[INFO]     at java.lang.ClassLoader.defineClass (ClassLoader.java:1017)
[INFO]     at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass (QuarkusClassLoader.java:543)
[INFO]     at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass (QuarkusClassLoader.java:501)
[INFO]     at io.vertx.core.impl.TaskQueue.execute (TaskQueue.java:157)
[INFO]     at io.vertx.core.impl.ContextImpl.internalExecuteBlocking (ContextImpl.java:217)
[INFO]     at io.vertx.core.impl.ContextImpl.executeBlocking (ContextImpl.java:189)
[INFO]     at io.vertx.core.impl.ContextImpl.executeBlockingInternal (ContextImpl.java:111)
[INFO]     at io.vertx.core.impl.ContextInternal.executeBlockingInternal (ContextInternal.java:152)
[INFO]     at io.vertx.core.impl.VertxInternal.executeBlockingInternal (VertxInternal.java:196)
[INFO]     at io.vertx.core.impl.VertxImpl.deleteCacheDirAndShutdown (VertxImpl.java:959)
[INFO]     at io.vertx.core.impl.VertxImpl.lambda$null$11 (VertxImpl.java:747)
[INFO]     at io.vertx.core.impl.VertxImpl.lambda$closeClusterManager$9 (VertxImpl.java:713)
[INFO]     at io.vertx.core.impl.future.FutureImpl$4.onSuccess (FutureImpl.java:176)
[INFO]     at io.vertx.core.impl.future.FutureBase.emitSuccess (FutureBase.java:66)
[INFO]     at io.vertx.core.impl.future.FutureImpl.addListener (FutureImpl.java:231)
[INFO]     at io.vertx.core.impl.future.FutureImpl.onComplete (FutureImpl.java:199)
[INFO]     at io.vertx.core.impl.VertxImpl.closeClusterManager (VertxImpl.java:708)
[INFO]     at io.vertx.core.impl.VertxImpl.lambda$null$12 (VertxImpl.java:745)
[INFO]     at io.vertx.core.impl.future.FutureImpl$4.onSuccess (FutureImpl.java:176)
[INFO]     at io.vertx.core.impl.future.FutureBase.emitSuccess (FutureBase.java:66)
[INFO]     at io.vertx.core.impl.future.FutureImpl.addListener (FutureImpl.java:231)
[INFO]     at io.vertx.core.impl.future.FutureImpl.onComplete (FutureImpl.java:199)
[INFO]     at io.vertx.core.impl.VertxImpl.lambda$null$13 (VertxImpl.java:744)
[INFO]     at io.vertx.core.impl.resolver.DefaultResolverProvider.close (DefaultResolverProvider.java:34)
[INFO]     at io.vertx.core.impl.AddressResolver.close (AddressResolver.java:118)
[INFO]     at io.vertx.core.impl.VertxImpl.lambda$null$14 (VertxImpl.java:741)
[INFO]     at io.vertx.core.impl.future.FutureImpl$4.onSuccess (FutureImpl.java:176)
[INFO]     at io.vertx.core.impl.future.FutureBase.emitSuccess (FutureBase.java:66)
[INFO]     at io.vertx.core.impl.future.FutureImpl.addListener (FutureImpl.java:231)
[INFO]     at io.vertx.core.impl.future.FutureImpl.onComplete (FutureImpl.java:199)
[INFO]     at io.vertx.core.impl.VertxImpl.lambda$null$15 (VertxImpl.java:740)
[INFO]     at io.vertx.core.impl.ContextImpl.emit (ContextImpl.java:328)
[INFO]     at io.vertx.core.impl.ContextImpl.lambda$emit$5 (ContextImpl.java:335)
[INFO]     at io.netty.util.concurrent.AbstractEventExecutor.runTask (AbstractEventExecutor.java:173)
[INFO]     at io.netty.util.concurrent.AbstractEventExecutor.safeExecute (AbstractEventExecutor.java:166)
[INFO]     at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks (SingleThreadEventExecutor.java:469)
[INFO]     at io.netty.channel.nio.NioEventLoop.run (NioEventLoop.java:569)
[INFO]     at io.netty.util.concurrent.SingleThreadEventExecutor$4.run (SingleThreadEventExecutor.java:994)
[INFO]     at io.netty.util.internal.ThreadExecutorMap$2.run (ThreadExecutorMap.java:74)
[INFO]     at io.netty.util.concurrent.FastThreadLocalRunnable.run (FastThreadLocalRunnable.java:30)
[INFO]     at java.lang.Thread.run (Thread.java:840)
  1. io.quarkus.netty.deployment.JBossNettyLoggerFactory$JBossNettyInternalLogger.warn
2024-06-16 12:04:49,958 WARN [io.net.cha.nio.NioEventLoop] (vert.x-eventloop-thread-0) Unexpected exception in the selector loop.: java.lang.IllegalStateException: Class loader Augmentation Class Loader: PROD for metrics has been closed and may not be accessed anymore
at io.quarkus.bootstrap.classloading.QuarkusClassLoader.ensureOpen(QuarkusClassLoader.java:723)
at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:499)
at io.quarkus.netty.deployment.JBossNettyLoggerFactory$JBossNettyInternalLogger.warn(JBossNettyLoggerFactory.java:199)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:168)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:1583)
  1. io.netty.util.concurrent.GlobalEventExecutor.startThread
[INFO] java.lang.IllegalStateException: Class loader Augmentation Class Loader: PROD for kubernetes-docker-build-and-deploy-statefulset-0.1-SNAPSHOT has been closed and may not be accessed anymore
[INFO] at io.quarkus.bootstrap.classloading.QuarkusClassLoader.ensureOpen (QuarkusClassLoader.java:723)
[INFO] at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass (QuarkusClassLoader.java:499)
[INFO] at io.netty.util.concurrent.GlobalEventExecutor.startThread (GlobalEventExecutor.java:239)
[INFO] at io.netty.util.concurrent.GlobalEventExecutor.execute0 (GlobalEventExecutor.java:227)
[INFO] at io.netty.util.concurrent.GlobalEventExecutor.execute (GlobalEventExecutor.java:221)
[INFO] at io.netty.util.concurrent.DefaultPromise.safeExecute (DefaultPromise.java:862)
[INFO] at io.netty.util.concurrent.DefaultPromise.notifyListeners (DefaultPromise.java:500)
[INFO] at io.netty.util.concurrent.DefaultPromise.setValue0 (DefaultPromise.java:636)
[INFO] at io.netty.util.concurrent.DefaultPromise.setSuccess0 (DefaultPromise.java:625)
[INFO] at io.netty.util.concurrent.DefaultPromise.setSuccess (DefaultPromise.java:97)
[INFO] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run (SingleThreadEventExecutor.java:1056)
[INFO] at io.netty.util.internal.ThreadExecutorMap$2.run (ThreadExecutorMap.java:74)
[INFO] at io.netty.util.concurrent.FastThreadLocalRunnable.run (FastThreadLocalRunnable.java:30)

And related:

 2024-06-16 12:04:07,527 ERROR [io.net.uti.con.Def.rejectedExecution] (vert.x-acceptor-thread-0) Failed to submit a listener notification task. Event loop shut down?: java.lang.IllegalArgumentException: io.netty.util.concurrent
	at java.base/java.lang.ClassLoader.definePackage(ClassLoader.java:2215)
	at io.quarkus.bootstrap.classloading.QuarkusClassLoader.definePackage(QuarkusClassLoader.java:576)
	at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:542)
	at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:501)
	at io.netty.util.concurrent.GlobalEventExecutor.startThread(GlobalEventExecutor.java:239)
	at io.netty.util.concurrent.GlobalEventExecutor.execute0(GlobalEventExecutor.java:227)
	at io.netty.util.concurrent.GlobalEventExecutor.execute(GlobalEventExecutor.java:221)
	at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:862)
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:500)
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
	at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625)
	at io.netty.util.concurrent.DefaultPromise.setSuccess(DefaultPromise.java:97)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:1056)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:840)
  1. io.micrometer.core.instrument.binder.jvm.JvmHeapPressureMetrics
java.lang.IllegalStateException: Class loader Quarkus Base Runtime ClassLoader: TEST for QuarkusTestNested1WithCommonCaseTest (QuarkusTest) has been closed and may not be accessed anymore
at io.quarkus.bootstrap.classloading.QuarkusClassLoader.ensureOpen(QuarkusClassLoader.java:723)
at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:499)
at io.micrometer.core.instrument.binder.jvm.JvmHeapPressureMetrics.lambda$monitor$3(JvmHeapPressureMetrics.java:116)
at java.management/sun.management.NotificationEmitterSupport.sendNotification(NotificationEmitterSupport.java:155)
at jdk.management/com.sun.management.internal.GarbageCollectorExtImpl.createGCNotification(GarbageCollectorExtImpl.java:115)
  1. Instrumentation reload test
2024-06-16T15:46:48.9763012Z 2024-06-16 15:46:48,905 INFO  [io.qua.dep.dev.fil.StaticFileManager] (vert.x-worker-thread-1) Ignoring module-info.java in dev mode, set the `quarkus.live-reload.ignore-module-info` property to `false` in your project descriptor (`pom.xml` or `build.gradle`) to disable this behavior.
2024-06-16T15:46:49.1767168Z 2024-06-16 15:46:49,081 ERROR [io.qua.dep.dev.RuntimeUpdatesProcessor] (vert.x-worker-thread-1) Failed to replace classes via instrumentation: java.lang.IllegalStateException: Class loader Deployment Class Loader: DEV for acme-1.0-SNAPSHOT has been closed and may not be accessed anymore
2024-06-16T15:46:49.1770682Z 	at io.quarkus.bootstrap.classloading.QuarkusClassLoader.ensureOpen(QuarkusClassLoader.java:727)
2024-06-16T15:46:49.1772859Z 	at io.quarkus.bootstrap.classloading.QuarkusClassLoader.getElementsWithResource(QuarkusClassLoader.java:608)
2024-06-16T15:46:49.1775071Z 	at io.quarkus.deployment.steps.ClassTransformingBuildStep$1.apply(ClassTransformingBuildStep.java:151)
2024-06-16T15:46:49.1777149Z 	at io.quarkus.deployment.steps.ClassTransformingBuildStep$1.apply(ClassTransformingBuildStep.java:126)
2024-06-16T15:46:49.1779287Z 	at io.quarkus.deployment.steps.ClassTransformingBuildStep.transform(ClassTransformingBuildStep.java:74)
2024-06-16T15:46:49.1781372Z 	at io.quarkus.deployment.dev.IsolatedDevModeMain$2.apply(IsolatedDevModeMain.java:250)
2024-06-16T15:46:49.1782968Z 	at io.quarkus.deployment.dev.IsolatedDevModeMain$2.apply(IsolatedDevModeMain.java:247)
2024-06-16T15:46:49.1784749Z 	at io.quarkus.deployment.dev.RuntimeUpdatesProcessor.doScan(RuntimeUpdatesProcessor.java:502)
2024-06-16T15:46:49.1786439Z 	at io.quarkus.deployment.dev.RuntimeUpdatesProcessor.doScan(RuntimeUpdatesProcessor.java:448)
2024-06-16T15:46:49.1788237Z 	at io.quarkus.vertx.http.runtime.devmode.VertxHttpHotReplacementSetup$6.call(VertxHttpHotReplacementSetup.java:161)
2024-06-16T15:46:49.1790223Z 	at io.quarkus.vertx.http.runtime.devmode.VertxHttpHotReplacementSetup$6.call(VertxHttpHotReplacementSetup.java:148)
2024-06-16T15:46:49.1792376Z 	at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$0(ContextImpl.java:178)
2024-06-16T15:46:49.1793816Z 	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:279)
2024-06-16T15:46:49.1795247Z 	at io.vertx.core.impl.ContextImpl.lambda$internalExecuteBlocking$2(ContextImpl.java:210)
2024-06-16T15:46:49.1796526Z 	at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
2024-06-16T15:46:49.1797817Z 	at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2516)
2024-06-16T15:46:49.1799430Z 	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2495)
2024-06-16T15:46:49.1814784Z 	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1521)
2024-06-16T15:46:49.1816132Z 	at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
2024-06-16T15:46:49.1817469Z 	at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
2024-06-16T15:46:49.1818968Z 	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
2024-06-16T15:46:49.1820118Z 	at java.base/java.lang.Thread.run(Thread.java:840)

This is due to this piece of code in ClassTransformingBuildStep (usage of cl):

        shutdown.addCloseTask(ClassTransformingBuildStep::reset, true);
        lastTransformers = new BiFunction<String, byte[], byte[]>() {
            @Override
            public byte[] apply(String className, byte[] originalBytes) {

                List<BytecodeTransformerBuildItem> classTransformers = bytecodeTransformers.get(className);
                if (classTransformers == null) {
                    return originalBytes;
                }
                boolean continueOnFailure = classTransformers.stream()
                        .filter(a -> !a.isContinueOnFailure())
                        .findAny().isEmpty();
                List<BiFunction<String, ClassVisitor, ClassVisitor>> visitors = classTransformers.stream()
                        .sorted(Comparator.comparingInt(BytecodeTransformerBuildItem::getPriority))
                        .map(BytecodeTransformerBuildItem::getVisitorFunction)
                        .filter(Objects::nonNull)
                        .collect(Collectors.toList());
                List<BiFunction<String, byte[], byte[]>> preVisitFunctions = classTransformers.stream()
                        .sorted(Comparator.comparingInt(BytecodeTransformerBuildItem::getPriority))
                        .map(BytecodeTransformerBuildItem::getInputTransformer)
                        .filter(Objects::nonNull)
                        .collect(Collectors.toList());
                ClassLoader old = Thread.currentThread().getContextClassLoader();
                try {
                    Thread.currentThread().setContextClassLoader(transformCl);
                    String classFileName = fromClassNameToResourceName(className);
                    List<ClassPathElement> archives = cl.getElementsWithResource(classFileName);
  1. Issues in DevUIGrpcSmokeTest

Hard to figure out what's going wrong here as we don't have the stacktraces handy.

  1. RESTEasy/Config in dev mode

✅ This one has been solved by a follow up commit.

2024-06-16T15:34:06.2997052Z 2024-06-16 15:34:05,500 ERROR [io.qua.ver.htt.run.QuarkusErrorHandler] (executor-thread-1) HTTP Request to /max-body-size failed, error id: a52a04c2-20ea-427c-8809-68804e849435-1 [Error Occurred After Shutdown]: java.lang.ExceptionInInitializerError
2024-06-16T15:34:06.3000384Z 	at org.jboss.resteasy.core.providerfactory.ResteasyProviderFactoryImpl.registerBuiltin(ResteasyProviderFactoryImpl.java:201)
2024-06-16T15:34:06.3002613Z 	at org.jboss.resteasy.spi.ResteasyProviderFactory.getInstance(ResteasyProviderFactory.java:115)
2024-06-16T15:34:06.3004440Z 	at io.quarkus.resteasy.runtime.standalone.RequestDispatcher.service(RequestDispatcher.java:91)
2024-06-16T15:34:06.3006078Z 	at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler.dispatch(VertxRequestHandler.java:151)
2024-06-16T15:34:06.3007707Z 	at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler$1.run(VertxRequestHandler.java:97)
2024-06-16T15:34:06.3009191Z 	at io.quarkus.vertx.core.runtime.VertxCoreRecorder$14.runWith(VertxCoreRecorder.java:599)
2024-06-16T15:34:06.3010602Z 	at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2516)
2024-06-16T15:34:06.3011953Z 	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2495)
2024-06-16T15:34:06.3013305Z 	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1521)
2024-06-16T15:34:06.3014580Z 	at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
2024-06-16T15:34:06.3015847Z 	at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
2024-06-16T15:34:06.3028336Z 	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
2024-06-16T15:34:06.3029607Z 	at java.base/java.lang.Thread.run(Thread.java:840)
2024-06-16T15:34:06.3031526Z Caused by: java.lang.IllegalStateException: Class loader Quarkus Runtime ClassLoader: DEV for testChunkedLargeRequest() (QuarkusDevModeTest) restart no:0 has been closed and may not be accessed anymore
2024-06-16T15:34:06.3034400Z 	at io.quarkus.bootstrap.classloading.QuarkusClassLoader.ensureOpen(QuarkusClassLoader.java:727)
2024-06-16T15:34:06.3036325Z 	at io.quarkus.bootstrap.classloading.QuarkusClassLoader.getResources(QuarkusClassLoader.java:226)
2024-06-16T15:34:06.3038011Z 	at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.nextProviderClass(ServiceLoader.java:1203)
2024-06-16T15:34:06.3176043Z 	at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1228)
2024-06-16T15:34:06.3177925Z 	at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1273)
2024-06-16T15:34:06.3179227Z 	at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1309)
2024-06-16T15:34:06.3180271Z 	at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1393)
2024-06-16T15:34:06.3181646Z 	at io.smallrye.config.SmallRyeConfigProviderResolver.getFactoryFor(SmallRyeConfigProviderResolver.java:102)
2024-06-16T15:34:06.3183298Z 	at io.smallrye.config.SmallRyeConfigProviderResolver.getConfig(SmallRyeConfigProviderResolver.java:78)
2024-06-16T15:34:06.3184881Z 	at io.smallrye.config.SmallRyeConfigProviderResolver.getConfig(SmallRyeConfigProviderResolver.java:66)
2024-06-16T15:34:06.3186265Z 	at org.eclipse.microprofile.config.ConfigProvider.getConfig(ConfigProvider.java:85)
2024-06-16T15:34:06.3188279Z 	at org.jboss.resteasy.microprofile.config.ConfigConfiguration.<init>(ConfigConfiguration.java:36)
2024-06-16T15:34:06.3189993Z 	at org.jboss.resteasy.microprofile.config.ConfigConfigurationFactory.getConfiguration(ConfigConfigurationFactory.java:32)
2024-06-16T15:34:06.3191645Z 	at org.jboss.resteasy.plugins.providers.RegisterBuiltin$5.run(RegisterBuiltin.java:178)
2024-06-16T15:34:06.3193059Z 	at org.jboss.resteasy.plugins.providers.RegisterBuiltin$5.run(RegisterBuiltin.java:175)
2024-06-16T15:34:06.3194411Z 	at java.base/java.security.AccessController.doPrivileged(AccessController.java:318)
2024-06-16T15:34:06.3195855Z 	at org.jboss.resteasy.plugins.providers.RegisterBuiltin.isGZipEnabled(RegisterBuiltin.java:175)
2024-06-16T15:34:06.3197355Z 	at org.jboss.resteasy.plugins.providers.RegisterBuiltin.<clinit>(RegisterBuiltin.java:37)
  1. Wiremock

✅ This one has been solved by a follow up commit.

Got it with Stork:

2024-06-16T16:23:09.0013999Z 	Suppressed: java.lang.IllegalStateException: Class loader Quarkus Base Runtime ClassLoader: TEST for StorkResponseTimeLoadBalancerTest (QuarkusUnitTest) has been closed and may not be accessed anymore
2024-06-16T16:23:09.0016332Z 		at io.quarkus.bootstrap.classloading.QuarkusClassLoader.ensureOpen(QuarkusClassLoader.java:727)
2024-06-16T16:23:09.0017924Z 		at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:503)
2024-06-16T16:23:09.0019360Z 		at wiremock.org.eclipse.jetty.io.ManagedSelector.doStop(ManagedSelector.java:135)
2024-06-16T16:23:09.0020806Z 		at wiremock.org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:132)
2024-06-16T16:23:09.0022389Z 		at wiremock.org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:182)
2024-06-16T16:23:09.0023989Z 		at wiremock.org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:205)
2024-06-16T16:23:09.0025461Z 		at wiremock.org.eclipse.jetty.io.SelectorManager.doStop(SelectorManager.java:280)
2024-06-16T16:23:09.0026870Z 		at wiremock.org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:132)
2024-06-16T16:23:09.0028472Z 		at wiremock.org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:182)
2024-06-16T16:23:09.0030097Z 		at wiremock.org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:205)
2024-06-16T16:23:09.0031630Z 		at wiremock.org.eclipse.jetty.server.AbstractConnector.doStop(AbstractConnector.java:425)
2024-06-16T16:23:09.0033181Z 		at wiremock.org.eclipse.jetty.server.AbstractNetworkConnector.doStop(AbstractNetworkConnector.java:82)
2024-06-16T16:23:09.0034923Z 		at wiremock.org.eclipse.jetty.server.ServerConnector.doStop(ServerConnector.java:240)
2024-06-16T16:23:09.0036442Z 		at wiremock.org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:132)
2024-06-16T16:23:09.0037797Z 		at wiremock.org.eclipse.jetty.server.Server.doStop(Server.java:506)

Also got it with REST Client:

2024-06-16T16:26:18.4058720Z ##[error]Exception in thread "Thread-3842" MultiException[java.lang.IllegalStateException: Class loader Quarkus Base Runtime ClassLoader: TEST for TrustAllTest (QuarkusUnitTest) has been closed and may not be accessed anymore, java.lang.IllegalStateException: Class loader Quarkus Base Runtime ClassLoader: TEST for TrustAllTest (QuarkusUnitTest) has been closed and may not be accessed anymore, java.lang.IllegalStateException: Class loader Quarkus Base Runtime ClassLoader: TEST for TrustAllTest (QuarkusUnitTest) has been closed and may not be accessed anymore, java.lang.IllegalStateException: Class loader Quarkus Base Runtime ClassLoader: TEST for TrustAllTest (QuarkusUnitTest) has been closed and may not be accessed anymore, java.lang.IllegalStateException: Class loader Quarkus Base Runtime ClassLoader: TEST for TrustAllTest (QuarkusUnitTest) has been closed and may not be accessed anymore, java.lang.IllegalStateException: Class loader Quarkus Base Runtime ClassLoader: TEST for TrustAllTest (QuarkusUnitTest) has been closed and may not be accessed anymore, java.lang.IllegalStateException: Class loader Quarkus Base Runtime ClassLoader: TEST for TrustAllTest (QuarkusUnitTest) has been closed and may not be accessed anymore]
2024-06-16T16:26:18.4070116Z 	at wiremock.org.eclipse.jetty.util.MultiException.ifExceptionThrow(MultiException.java:117)
2024-06-16T16:26:18.4071515Z 	at wiremock.org.eclipse.jetty.server.Server.doStop(Server.java:531)
2024-06-16T16:26:18.4072926Z 	at wiremock.org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:132)
2024-06-16T16:26:18.4074522Z 	at com.github.tomakehurst.wiremock.jetty.JettyHttpServer.stop(JettyHttpServer.java:178)
2024-06-16T16:26:18.4075921Z 	at com.github.tomakehurst.wiremock.WireMockServer.stop(WireMockServer.java:182)
2024-06-16T16:26:18.4077330Z 	at com.github.tomakehurst.wiremock.WireMockServer.lambda$shutdown$3(WireMockServer.java:215)
2024-06-16T16:26:18.4078476Z 	at java.base/java.lang.Thread.run(Thread.java:840)
2024-06-16T16:26:18.4080249Z 	Suppressed: java.lang.IllegalStateException: Class loader Quarkus Base Runtime ClassLoader: TEST for TrustAllTest (QuarkusUnitTest) has been closed and may not be accessed anymore
2024-06-16T16:26:18.4082406Z 		at io.quarkus.bootstrap.classloading.QuarkusClassLoader.ensureOpen(QuarkusClassLoader.java:727)
2024-06-16T16:26:18.4084348Z 		at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:503)
2024-06-16T16:26:18.4085850Z 		at wiremock.org.eclipse.jetty.io.ManagedSelector.doStop(ManagedSelector.java:135)
2024-06-16T16:26:18.4087331Z 		at wiremock.org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:132)
2024-06-16T16:26:18.4088957Z 		at wiremock.org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:182)
2024-06-16T16:26:18.4090689Z 		at wiremock.org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:205)
2024-06-16T16:26:18.4092202Z 		at wiremock.org.eclipse.jetty.io.SelectorManager.doStop(SelectorManager.java:280)
2024-06-16T16:26:18.4093675Z 		at wiremock.org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:132)
2024-06-16T16:26:18.4095285Z 		at wiremock.org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:182)
2024-06-16T16:26:18.4096931Z 		at wiremock.org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:205)
2024-06-16T16:26:18.4098521Z 		at wiremock.org.eclipse.jetty.server.AbstractConnector.doStop(AbstractConnector.java:425)
2024-06-16T16:26:18.4100470Z 		at wiremock.org.eclipse.jetty.server.AbstractNetworkConnector.doStop(AbstractNetworkConnector.java:82)
2024-06-16T16:26:18.4102089Z 		at wiremock.org.eclipse.jetty.server.ServerConnector.doStop(ServerConnector.java:240)
2024-06-16T16:26:18.4103614Z 		at wiremock.org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:132)
2024-06-16T16:26:18.4104900Z 		at wiremock.org.eclipse.jetty.server.Server.doStop(Server.java:506)
2024-06-16T16:26:18.4105674Z 		... 5 more
Copy link

quarkus-bot bot commented Jun 16, 2024

/cc @geoand (kubernetes,minikube), @iocanel (kubernetes)

@gsmet
Copy link
Member Author

gsmet commented Jun 16, 2024

/cc @dmlloyd @cescoffier I think your feedback will be invaluable here too.

@cescoffier
Copy link
Member

There is a hack iterating over Vert.x event loops to reset the TCCL in dev mode. That might be the problem.

@geoand
Copy link
Contributor

geoand commented Jun 17, 2024

#40942 (comment) might also be related

@dmlloyd
Copy link
Member

dmlloyd commented Jun 17, 2024

We are very cavalier about how we create and use class loaders. Since we don't have any kind of determinism around them, we end up with non-deterministic consequences. Throwing an exception is a good start, but troubleshooting these problems is going to be pretty tricky I suspect.

@HerrDerb
Copy link
Contributor

HerrDerb commented Jun 18, 2024

Can this cause classloader issues during QuarkusTest? Since shortly, we experience inconsistent classloader issues when running QuarkusTests within the scope of the actual test class, of which we are still confused.

java.util.concurrent.ExecutionException: java.lang.ClassCastException: class com.company.MyClass cannot be cast to class com.company.MyClass (com.company.MyClass  is in unnamed module of loader 'app'; com.company.MyClass  is in unnamed module of loader io.quarkus.bootstrap.classloading.QuarkusClassLoader @3aeba81)

@dmlloyd
Copy link
Member

dmlloyd commented Jun 18, 2024

Can this cause classloader issues during QuarkusTest? Since shortly, we experience inconsistent classloader issues when running QuarkusTests within the scope of the actual test class, of which we are still confused.

No, more likely you need #40906 for this one, but it's just a guess based on this small snippet. Once that PR is merged, if you still have an issue, then consider filing a bug or opening a discussion topic for it.

gsmet added a commit to gsmet/quarkus that referenced this issue Jun 25, 2024
When popping provider factory, what we actually want is pop it if
something has been pushed.
It is better handled by setting a boolean if something has been pushed.

This allows class loading issues when the class loader has been closed
and the request is still processing.

Related to quarkusio#41233
gsmet added a commit to gsmet/quarkus that referenced this issue Jun 25, 2024
And also avoid passing the CuratedApplication here.
The problem was noticed when trying to log something in the
QuarkusClassLoader constructor.

Related to quarkusio#41233
gsmet added a commit to gsmet/quarkus that referenced this issue Jun 25, 2024
- The TestResourceManager is instantiated using the runtime class loader
  so we need to make sure we close it before we close the runtime class
  loader. Otherwise we can get into trouble if we need to load some
  additional classes.
- Also make sure that it is only closed once. It used to be closed once
  when the application is stopped and then again when the
  QuarkusTestExtensionState was closed.

Related to quarkusio#41233
gsmet added a commit to gsmet/quarkus that referenced this issue Jun 25, 2024
Currently, we pass the CL to something that will close it, rather than
controlling the lifecycle in the creator method.

This is an issue in native ITs:

java.lang.RuntimeException: java.lang.IllegalStateException: This class loader has been closed
	at io.quarkus.test.junit.QuarkusIntegrationTestExtension.throwBootFailureException(QuarkusIntegrationTestExtension.java:373)
	at io.quarkus.test.junit.QuarkusIntegrationTestExtension.beforeEach(QuarkusIntegrationTestExtension.java:117)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
Caused by: java.lang.IllegalStateException: This class loader has been closed
	at io.quarkus.bootstrap.classloading.QuarkusClassLoader.ensureOpen(QuarkusClassLoader.java:716)
	at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:495)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:467)
	at io.quarkus.runner.bootstrap.AugmentActionImpl.performCustomBuild(AugmentActionImpl.java:158)
	at io.quarkus.test.junit.IntegrationTestUtil.handleDevServices(IntegrationTestUtil.java:297)
	at io.quarkus.test.junit.QuarkusIntegrationTestExtension.doProcessStart(QuarkusIntegrationTestExtension.java:199)
	at io.quarkus.test.junit.QuarkusIntegrationTestExtension.ensureStarted(QuarkusIntegrationTestExtension.java:169)
	at io.quarkus.test.junit.QuarkusIntegrationTestExtension.beforeAll(QuarkusIntegrationTestExtension.java:130)

Related to quarkusio#41233
holly-cummins pushed a commit to holly-cummins/quarkus that referenced this issue Jul 31, 2024
Currently, we pass the CL to something that will close it, rather than
controlling the lifecycle in the creator method.

This is an issue in native ITs:

java.lang.RuntimeException: java.lang.IllegalStateException: This class loader has been closed
	at io.quarkus.test.junit.QuarkusIntegrationTestExtension.throwBootFailureException(QuarkusIntegrationTestExtension.java:373)
	at io.quarkus.test.junit.QuarkusIntegrationTestExtension.beforeEach(QuarkusIntegrationTestExtension.java:117)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
Caused by: java.lang.IllegalStateException: This class loader has been closed
	at io.quarkus.bootstrap.classloading.QuarkusClassLoader.ensureOpen(QuarkusClassLoader.java:716)
	at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:495)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:467)
	at io.quarkus.runner.bootstrap.AugmentActionImpl.performCustomBuild(AugmentActionImpl.java:158)
	at io.quarkus.test.junit.IntegrationTestUtil.handleDevServices(IntegrationTestUtil.java:297)
	at io.quarkus.test.junit.QuarkusIntegrationTestExtension.doProcessStart(QuarkusIntegrationTestExtension.java:199)
	at io.quarkus.test.junit.QuarkusIntegrationTestExtension.ensureStarted(QuarkusIntegrationTestExtension.java:169)
	at io.quarkus.test.junit.QuarkusIntegrationTestExtension.beforeAll(QuarkusIntegrationTestExtension.java:130)

Related to quarkusio#41233
holly-cummins pushed a commit to holly-cummins/quarkus that referenced this issue Jul 31, 2024
- The TestResourceManager is instantiated using the runtime class loader
  so we need to make sure we close it before we close the runtime class
  loader. Otherwise we can get into trouble if we need to load some
  additional classes.
- Also make sure that it is only closed once. It used to be closed once
  when the application is stopped and then again when the
  QuarkusTestExtensionState was closed.

Related to quarkusio#41233
holly-cummins pushed a commit to holly-cummins/quarkus that referenced this issue Jul 31, 2024
When popping provider factory, what we actually want is pop it if
something has been pushed.
It is better handled by setting a boolean if something has been pushed.

This allows class loading issues when the class loader has been closed
and the request is still processing.

Related to quarkusio#41233
@geoand
Copy link
Contributor

geoand commented Aug 8, 2024

4 has been resolved in #42388

danielsoro pushed a commit to danielsoro/quarkus that referenced this issue Sep 20, 2024
Currently, we pass the CL to something that will close it, rather than
controlling the lifecycle in the creator method.

This is an issue in native ITs:

java.lang.RuntimeException: java.lang.IllegalStateException: This class loader has been closed
	at io.quarkus.test.junit.QuarkusIntegrationTestExtension.throwBootFailureException(QuarkusIntegrationTestExtension.java:373)
	at io.quarkus.test.junit.QuarkusIntegrationTestExtension.beforeEach(QuarkusIntegrationTestExtension.java:117)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
Caused by: java.lang.IllegalStateException: This class loader has been closed
	at io.quarkus.bootstrap.classloading.QuarkusClassLoader.ensureOpen(QuarkusClassLoader.java:716)
	at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:495)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:467)
	at io.quarkus.runner.bootstrap.AugmentActionImpl.performCustomBuild(AugmentActionImpl.java:158)
	at io.quarkus.test.junit.IntegrationTestUtil.handleDevServices(IntegrationTestUtil.java:297)
	at io.quarkus.test.junit.QuarkusIntegrationTestExtension.doProcessStart(QuarkusIntegrationTestExtension.java:199)
	at io.quarkus.test.junit.QuarkusIntegrationTestExtension.ensureStarted(QuarkusIntegrationTestExtension.java:169)
	at io.quarkus.test.junit.QuarkusIntegrationTestExtension.beforeAll(QuarkusIntegrationTestExtension.java:130)

Related to quarkusio#41233
danielsoro pushed a commit to danielsoro/quarkus that referenced this issue Sep 20, 2024
- The TestResourceManager is instantiated using the runtime class loader
  so we need to make sure we close it before we close the runtime class
  loader. Otherwise we can get into trouble if we need to load some
  additional classes.
- Also make sure that it is only closed once. It used to be closed once
  when the application is stopped and then again when the
  QuarkusTestExtensionState was closed.

Related to quarkusio#41233
danielsoro pushed a commit to danielsoro/quarkus that referenced this issue Sep 20, 2024
When popping provider factory, what we actually want is pop it if
something has been pushed.
It is better handled by setting a boolean if something has been pushed.

This allows class loading issues when the class loader has been closed
and the request is still processing.

Related to quarkusio#41233
@aureamunoz
Copy link
Member

It seems this one has been already resolved, Can we close it? @gsmet

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants