Narayana transaction recovery stops work in DEV mode after hot reload #35839

michalvavrik · 2023-09-10T20:33:52Z

Describe the bug

When I change my application in DEV mode and the app is reload, Narayana keeps using the old (previous) Datasource connection even though pool has been closed and new one was created. Details in here #35242 (comment)

Expected behavior

After hot reload, my app still works.

Actual behavior

Warning is logged and periodic recovery doesn't work

2023-08-07 18:41:54,145 WARN  [com.arj.ats.arjuna] (Transaction Expired Entry Monitor) ARJUNA012251: allObjUids caught exception: java.sql.SQLException: This pool is closed and does not handle any more connections!
	at io.agroal.pool.ConnectionPool.beforeAcquire(ConnectionPool.java:209)
	at io.agroal.pool.ConnectionPool.getConnection(ConnectionPool.java:235)
	at io.agroal.pool.DataSource.getConnection(DataSource.java:86)
	at io.quarkus.narayana.jta.runtime.QuarkusDataSource.getConnection(QuarkusDataSource.java:38)
	at com.arjuna.ats.internal.arjuna.objectstore.jdbc.accessors.DirectDataSourceJDBCAccess.getConnection(DirectDataSourceJDBCAccess.java:27)
	at com.arjuna.ats.internal.arjuna.objectstore.jdbc.JDBCImple_driver.allObjUids(JDBCImple_driver.java:348)
	at com.arjuna.ats.internal.arjuna.objectstore.jdbc.JDBCStore.allObjUids(JDBCStore.java:169)
	at com.arjuna.ats.internal.arjuna.objectstore.jdbc.JDBCStore.allObjUids(JDBCStore.java:125)
	at com.arjuna.ats.internal.arjuna.recovery.ExpiredTransactionStatusManagerScanner.scan(ExpiredTransactionStatusManagerScanner.java:81)
	at com.arjuna.ats.internal.arjuna.recovery.ExpiredEntryMonitor.run(ExpiredEntryMonitor.java:171)

How to Reproduce?

Reproducer:

on this branch and inside quickstart added here Add Narayana Transaction Logs QuickStart quarkus-quickstarts#1311
quarkus dev
wait till service is fully started
force restart with s

Output of `uname -a` or `ver`

Fedora

Output of `java -version`

17

GraalVM version (if different from Java)

23

Quarkus version or git rev

999-SNAPSHOT

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

Apache Maven 3.9.3

Additional information

No response

The text was updated successfully, but these errors were encountered:

michalvavrik · 2023-09-10T20:36:52Z

/cc @zhfeng @mmusgrov

michalvavrik · 2024-04-16T21:54:17Z

Is this solved or on your radar @zhfeng @mmusgrov ?

mmusgrov · 2024-04-22T07:57:25Z

@michalvavrik I don't recall ever investigating it. You raised the issue so could you not retest it? If not then the transaction team are quite busy: @marcosgopen or @jmfinelli are either of you able to provide Michal with any input on this issue.

michalvavrik · 2024-04-22T08:00:23Z

@michalvavrik I don't recall ever investigating it. You raised the issue so could you not retest it? If not then the transaction team are quite busy: @marcosgopen or @jmfinelli are either of you able to provide Michal with any input on this issue.

copy that, thanks; I'll re-run the test later this week

marcosgopen · 2024-04-22T09:50:27Z

Sure, I will have a look at it.

marcosgopen · 2024-04-23T15:20:58Z

Hi @michalvavrik , I tested your reproducer but I got an error:
[INFO] Running org.acme.quickstart.JdbcObjectStoreTest [ERROR] Tests run: 5, Failures: 0, Errors: 1, Skipped: 4, Time elapsed: 4.396 s <<< FAILURE! - in org.acme.quickstart.JdbcObjectStoreTest [ERROR] org.acme.quickstart.JdbcObjectStoreTest.testTransactionsLogWithInjectedUserTransaction Time elapsed: 0.015 s <<< ERROR! java.lang.RuntimeException: java.lang.RuntimeException: Failed to load steps from class io.quarkus.resteasy.reactive.server.deployment.ResteasyReactiveProcessor at io.quarkus.test.junit.QuarkusTestExtension.throwBootFailureException(QuarkusTestExtension.java:643) at io.quarkus.test.junit.QuarkusTestExtension.interceptTestClassConstructor(QuarkusTestExtension.java:727) at java.base/java.util.Optional.orElseGet(Optional.java:364) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) Caused by: java.lang.RuntimeException: Failed to load steps from class io.quarkus.resteasy.reactive.server.deployment.ResteasyReactiveProcessor at io.quarkus.deployment.ExtensionLoader.loadStepsFrom(ExtensionLoader.java:164) at io.quarkus.deployment.QuarkusAugmentor.run(QuarkusAugmentor.java:107) at io.quarkus.runner.bootstrap.AugmentActionImpl.runAugment(AugmentActionImpl.java:333) at io.quarkus.runner.bootstrap.AugmentActionImpl.createInitialRuntimeApplication(AugmentActionImpl.java:254) at io.quarkus.runner.bootstrap.AugmentActionImpl.createInitialRuntimeApplication(AugmentActionImpl.java:60) at io.quarkus.test.junit.QuarkusTestExtension.doJavaStart(QuarkusTestExtension.java:224) at io.quarkus.test.junit.QuarkusTestExtension.ensureStarted(QuarkusTestExtension.java:610) at io.quarkus.test.junit.QuarkusTestExtension.beforeAll(QuarkusTestExtension.java:660) ... 1 more Caused by: java.lang.TypeNotPresentException: Type io.quarkus.vertx.http.deployment.EagerSecurityInterceptorBuildItem not present at java.base/java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:298) at java.base/java.lang.reflect.Method.getGenericParameterTypes(Method.java:333) at java.base/java.lang.reflect.Executable.getAllGenericParameterTypes(Executable.java:317) at java.base/java.lang.reflect.Parameter.getParameterizedType(Parameter.java:202) at io.quarkus.deployment.ExtensionLoader.loadStepsFromClass(ExtensionLoader.java:493) at io.quarkus.deployment.ExtensionLoader.loadStepsFrom(ExtensionLoader.java:162) ... 8 more Caused by: java.lang.ClassNotFoundException: io.quarkus.vertx.http.deployment.EagerSecurityInterceptorBuildItem at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525) at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:518) at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:468) at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:467) ... 14 more
Are you also experiencing that? I have just built quarkus main branch and there might be 'breaking' changes. If you can still reproduce the issue, do you mind specify the 'git rev' of the quarkus you are using (e.g. the commit sha)? Thanks

michalvavrik · 2024-04-27T18:01:57Z

sorry @marcosgopen , I had very busy week, sure, there were breaking changes in the meanwhile and I completely forgot about QS PR. Should probably rebase it and resolve conflicts. I'll fix the reproducer next week, thanks so far.

gsmet · 2024-08-19T12:45:34Z

It would be interesting to see if this is still an issue.

marcosgopen · 2024-09-09T09:29:47Z

@michalvavrik could you reproduce the issue with a more recent version of quarkus?

michalvavrik · 2024-09-16T20:23:46Z

Hey, I'll answer later this week.

michalvavrik · 2024-09-22T20:00:05Z

So, I have tried and I still can reproduce the original issue. Here is how:

git clone [email protected]:michalvavrik/quarkus-quickstarts.git -b feature/add-narayana-transaction-logs-qs
cd quarkus-quickstarts/narayana-transaction-logs-quickstart
docker-compose up
quarkus dev

Wait till Quarkus ready, then type 's', maybe wait again and try thrice. I am not quite sure how many times I have tried s as I had a long day. Here are logs I got:

[INFO] Invoking compiler:3.1:testCompile (default-testCompile) @ narayana-transaction-logs-quickstart
[INFO] Nothing to compile - all classes are up to date
Listening for transport dt_socket at address: 5005
__  ____  __  _____   ___  __ ____  ______ 
 --/ __ \/ / / / _ | / _ \/ //_/ / / / __/ 
 -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \   
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/   
2024-09-22 21:57:58,602 INFO  [io.quarkus] (Quarkus Main Thread) narayana-transaction-logs-quickstart 1.0.0-SNAPSHOT on JVM (powered by Quarkus 999-SNAPSHOT) started in 2.090s. Listening on: http://localhost:8080

2024-09-22 21:57:58,605 INFO  [io.quarkus] (Quarkus Main Thread) Profile dev activated. Live Coding activated.
2024-09-22 21:57:58,605 INFO  [io.quarkus] (Quarkus Main Thread) Installed features: [agroal, cdi, jdbc-postgresql, narayana-jta, rest, smallrye-context-propagation, vertx]
2024-09-22 21:58:01,087 INFO  [io.qua.dep.dev.RuntimeUpdatesProcessor] (Aesh InputStream Reader) Restarting as requested by the user.
2024-09-22 21:58:01,090 INFO  [com.arj.ats.jbossatx] (Quarkus Main Thread) ARJUNA032014: Stopping transaction recovery manager
2024-09-22 21:58:01,099 INFO  [io.quarkus] (Quarkus Main Thread) narayana-transaction-logs-quickstart stopped in 0.011s
__  ____  __  _____   ___  __ ____  ______ 
 --/ __ \/ / / / _ | / _ \/ //_/ / / / __/ 
 -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \   
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/   
2024-09-22 21:58:01,460 INFO  [io.quarkus] (Quarkus Main Thread) narayana-transaction-logs-quickstart 1.0.0-SNAPSHOT on JVM (powered by Quarkus 999-SNAPSHOT) started in 0.359s. Listening on: http://localhost:8080

2024-09-22 21:58:01,461 INFO  [io.quarkus] (Quarkus Main Thread) Profile dev activated. Live Coding activated.
2024-09-22 21:58:01,461 INFO  [io.quarkus] (Quarkus Main Thread) Installed features: [agroal, cdi, jdbc-postgresql, narayana-jta, rest, smallrye-context-propagation, vertx]
2024-09-22 21:58:01,462 INFO  [io.qua.dep.dev.RuntimeUpdatesProcessor] (Aesh InputStream Reader) Live reload total time: 0.377s 
2024-09-22 21:58:02,738 INFO  [io.qua.dep.dev.RuntimeUpdatesProcessor] (Aesh InputStream Reader) Restarting as requested by the user.
2024-09-22 21:58:02,743 INFO  [com.arj.ats.jbossatx] (Quarkus Main Thread) ARJUNA032014: Stopping transaction recovery manager
2024-09-22 21:58:02,748 INFO  [io.quarkus] (Quarkus Main Thread) narayana-transaction-logs-quickstart stopped in 0.008s
__  ____  __  _____   ___  __ ____  ______ 
 --/ __ \/ / / / _ | / _ \/ //_/ / / / __/ 
 -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \   
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/   
2024-09-22 21:58:03,051 WARN  [com.arj.ats.arjuna] (Transaction Expired Entry Monitor) ARJUNA012251: allObjUids caught exception: java.sql.SQLException: This pool is closed and does not handle any more connections!
	at io.agroal.pool.ConnectionPool.beforeAcquire(ConnectionPool.java:247)
	at io.agroal.pool.ConnectionPool.getConnection(ConnectionPool.java:273)
	at io.agroal.pool.DataSource.getConnection(DataSource.java:86)
	at io.agroal.api.AgroalDataSource_4TkLOYZRnQnK5Mzj2yarinAMQzA_Synthetic_ClientProxy.getConnection(Unknown Source)
	at io.quarkus.narayana.jta.runtime.QuarkusDataSource.getConnection(QuarkusDataSource.java:38)
	at com.arjuna.ats.internal.arjuna.objectstore.jdbc.accessors.DirectDataSourceJDBCAccess.getConnection(DirectDataSourceJDBCAccess.java:25)
	at com.arjuna.ats.internal.arjuna.objectstore.jdbc.JDBCImple_driver.allObjUids(JDBCImple_driver.java:321)
	at com.arjuna.ats.internal.arjuna.objectstore.jdbc.JDBCStore.allObjUids(JDBCStore.java:142)
	at com.arjuna.ats.internal.arjuna.objectstore.jdbc.JDBCStore.allObjUids(JDBCStore.java:98)
	at com.arjuna.ats.internal.arjuna.recovery.ExpiredTransactionStatusManagerScanner.scan(ExpiredTransactionStatusManagerScanner.java:54)
	at com.arjuna.ats.internal.arjuna.recovery.ExpiredEntryMonitor.run(ExpiredEntryMonitor.java:144)


2024-09-22 21:58:03,051 WARN  [com.arj.ats.arjuna] (Periodic Recovery) ARJUNA012251: allObjUids caught exception: java.sql.SQLException: This pool is closed and does not handle any more connections!
	at io.agroal.pool.ConnectionPool.beforeAcquire(ConnectionPool.java:247)
	at io.agroal.pool.ConnectionPool.getConnection(ConnectionPool.java:273)
	at io.agroal.pool.DataSource.getConnection(DataSource.java:86)
	at io.agroal.api.AgroalDataSource_4TkLOYZRnQnK5Mzj2yarinAMQzA_Synthetic_ClientProxy.getConnection(Unknown Source)
	at io.quarkus.narayana.jta.runtime.QuarkusDataSource.getConnection(QuarkusDataSource.java:38)
	at com.arjuna.ats.internal.arjuna.objectstore.jdbc.accessors.DirectDataSourceJDBCAccess.getConnection(DirectDataSourceJDBCAccess.java:25)
	at com.arjuna.ats.internal.arjuna.objectstore.jdbc.JDBCImple_driver.allObjUids(JDBCImple_driver.java:321)
	at com.arjuna.ats.internal.arjuna.objectstore.jdbc.JDBCStore.allObjUids(JDBCStore.java:142)
	at com.arjuna.ats.internal.arjuna.objectstore.jdbc.JDBCStore.allObjUids(JDBCStore.java:98)
	at com.arjuna.ats.internal.arjuna.recovery.AtomicActionRecoveryModule.periodicWorkFirstPass(AtomicActionRecoveryModule.java:63)
	at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:743)
	at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:355)


2024-09-22 21:58:03,053 WARN  [com.arj.ats.arjuna] (Periodic Recovery) ARJUNA012251: allObjUids caught exception: java.sql.SQLException: This pool is closed and does not handle any more connections!
	at io.agroal.pool.ConnectionPool.beforeAcquire(ConnectionPool.java:247)
	at io.agroal.pool.ConnectionPool.getConnection(ConnectionPool.java:273)
	at io.agroal.pool.DataSource.getConnection(DataSource.java:86)
	at io.agroal.api.AgroalDataSource_4TkLOYZRnQnK5Mzj2yarinAMQzA_Synthetic_ClientProxy.getConnection(Unknown Source)
	at io.quarkus.narayana.jta.runtime.QuarkusDataSource.getConnection(QuarkusDataSource.java:38)
	at com.arjuna.ats.internal.arjuna.objectstore.jdbc.accessors.DirectDataSourceJDBCAccess.getConnection(DirectDataSourceJDBCAccess.java:25)
	at com.arjuna.ats.internal.arjuna.objectstore.jdbc.JDBCImple_driver.allObjUids(JDBCImple_driver.java:321)
	at com.arjuna.ats.internal.arjuna.objectstore.jdbc.JDBCStore.allObjUids(JDBCStore.java:142)
	at com.arjuna.ats.internal.arjuna.objectstore.jdbc.JDBCStore.allObjUids(JDBCStore.java:98)
	at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkFirstPass(XARecoveryModule.java:198)
	at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkFirstPass(XARecoveryModule.java:167)
	at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:743)
	at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:355)


2024-09-22 21:58:03,054 WARN  [com.arj.ats.jta] (Periodic Recovery) ARJUNA016001: could not get all object Uids.
2024-09-22 21:58:03,054 INFO  [io.quarkus] (Quarkus Main Thread) narayana-transaction-logs-quickstart 1.0.0-SNAPSHOT on JVM (powered by Quarkus 999-SNAPSHOT) started in 0.303s. Listening on: http://localhost:8080
2024-09-22 21:58:03,054 INFO  [io.quarkus] (Quarkus Main Thread) Profile dev activated. Live Coding activated.
2024-09-22 21:58:03,055 INFO  [io.quarkus] (Quarkus Main Thread) Installed features: [agroal, cdi, jdbc-postgresql, narayana-jta, rest, smallrye-context-propagation, vertx]
2024-09-22 21:58:03,055 INFO  [io.qua.dep.dev.RuntimeUpdatesProcessor] (Aesh InputStream Reader) Live reload total time: 0.321s

marcosgopen · 2024-09-24T15:01:36Z

Thanks @michalvavrik for your reproducer, I menaged to reproduce it and I am working to understand what the problem is and solve it. I think that when the destroy method of QuarkusRecoveryService is executed we should also set to null the datasource of QuarkusDataSource. I'll keep you updated.

michalvavrik added kind/bug Something isn't working area/narayana Transactions / Narayana labels Sep 10, 2023

quarkus-bot bot added the area/devmode label Sep 10, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Narayana transaction recovery stops work in DEV mode after hot reload #35839

Narayana transaction recovery stops work in DEV mode after hot reload #35839

michalvavrik commented Sep 10, 2023

michalvavrik commented Sep 10, 2023

michalvavrik commented Apr 16, 2024

mmusgrov commented Apr 22, 2024

michalvavrik commented Apr 22, 2024

marcosgopen commented Apr 22, 2024

marcosgopen commented Apr 23, 2024

michalvavrik commented Apr 27, 2024 •

edited

Loading

gsmet commented Aug 19, 2024

marcosgopen commented Sep 9, 2024

michalvavrik commented Sep 16, 2024

michalvavrik commented Sep 22, 2024

marcosgopen commented Sep 24, 2024

Narayana transaction recovery stops work in DEV mode after hot reload #35839

Narayana transaction recovery stops work in DEV mode after hot reload #35839

Comments

michalvavrik commented Sep 10, 2023

Describe the bug

Expected behavior

Actual behavior

How to Reproduce?

Output of uname -a or ver

Output of java -version

GraalVM version (if different from Java)

Quarkus version or git rev

Build tool (ie. output of mvnw --version or gradlew --version)

Additional information

michalvavrik commented Sep 10, 2023

michalvavrik commented Apr 16, 2024

mmusgrov commented Apr 22, 2024

michalvavrik commented Apr 22, 2024

marcosgopen commented Apr 22, 2024

marcosgopen commented Apr 23, 2024

michalvavrik commented Apr 27, 2024 • edited Loading

gsmet commented Aug 19, 2024

marcosgopen commented Sep 9, 2024

michalvavrik commented Sep 16, 2024

michalvavrik commented Sep 22, 2024

marcosgopen commented Sep 24, 2024

Output of `uname -a` or `ver`

Output of `java -version`

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

michalvavrik commented Apr 27, 2024 •

edited

Loading