-
Notifications
You must be signed in to change notification settings - Fork 55
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Connection pool hangs after DB disconnect #107
Comments
The same with #68 ? |
I found a way to avoid this.
I notice
init with
the disconnected connection below to idle, but default max-idle-time is 30m, I tested with 2s, it works fine |
I suggest modify evictionPredicate, where connection is not valid, evict it |
Connection validation requires database I/O ( When the connection validation fails, then we invalidate the pooled object and propagate the error. Validation is guarded with |
If i set an acquire time of 2 seconds it changes behavior and i get this error on the timer queries: But the connections don't recover. I also tried adding a "SELET 1" validation query and same result. Using the max idle time of 2 seconds as suggested above does fix this sample app but if i change the recurring query timer to 1 second it breaks again. I.e. with the query happening in that idle window when the connection has not been invalidated due to idleness. Thanks. |
|
We've seen the exact same issue during a database restart. First all connections drop:
And then no new connections can be optained (using postgres)
Edit: |
This issue is risking one of our releases. Is there something we can do to help? |
We need to investigate why this is and we haven't gotten to the point yet where we had sufficient time to investigate the issue. Feel free to have a look into the flow and why some guards (like timeouts) don't seem to work. |
This may be of use - when we upgraded all our Spring/R2DBC libs the behavior changed. With the versions below what I see is that after the DB comes back up there are a couple more failures (which i think might remove bad connections form the pool) and then it reconnects. I've not isolated this to one lib or the other, but just upgrading the R2DBC libs doesn't seem to fix it alone.
|
Similar to @yossarian81 upgrading to above versions fixed the issue. The visible errors in io.r2dbc stayed the same, see below, and we suspect that the issue is caused somewhere in reactor libs, presumably reactor-pool. The call
|
Can you check from where the logging gets issued? That would be helpful to investigate whether failures get propagated and logged or whether the failure is just logged without affecting the application. |
Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue. |
Bug Report
Versions
Current Behavior
With the following steps and attached minimal test app I can repro a DB connection pool hang where killing and restarting the DB results in the app never reconnecting.
Stack trace
Table schema
Input Code
Steps to reproduce
Link to minimal test app reproducing issue:
https://github.com/yossarian81/r2dbc-issue
This app loads a minimal schema and some data, then it uses a Spring scheduled task to periodically read from the DB.
Steps:
The periodic data read application never recovers, it hangs when trying to read and with TRACE enabled the connection pool logs only:
"Obtaining new connection from the driver"
Tried many connection pool settings and I see the same behaviour.
Input Code
// your code here;
Expected behavior/code
Expectation is that the periodic task is able to read from the DB once the DB is restarted.
Possible Solution
Additional context
The text was updated successfully, but these errors were encountered: