-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Transaction is not always rolled back on client disconnect #7129
Comments
This is similar to mysql behaviour Session 1:
Session 2:
Session 1:
Session 1 is hung up for lock_wait_timeout. |
It is not. In your example you need to close your mysql client (Ctrl+D). Then the transaction in session 2 is immediately aborted and query in session 1 proceeds immediately. Session 1:
Session 2:
Ctrl+C in MySQL CLI does not close connection, it only aborts current query (sends KILL QUERY as opposed to KILL CONNECTION). |
I see that now. Ignore my earlier message. |
Overview of the Issue
We have an application that inserts a row into a table and allows 5 seconds for the transaction to complete. If the transaction is not completed after 5 seconds, the application closes and reopens the connection, and tries to insert the row again. If the application hits a duplicate key error, or a success, it moves on to the next record.
Occasionally, there is a situation where insert cannot complete within the timeout (e.g. if the row is locked by another transaction that is waiting for a semi sync ACK from replicas), which causes the application to disconnect, and retry after a back-off interval. We expect that upon disconnect, Vitess would roll back any transaction being executed inside the given connection. However, Vitess does not roll back such transactions, which causes retried inserts to pile-up in MySQL. This goes on until either
innodb_lock_wait_timeout
orqueryserver-config-transaction-timeout
expires and transactions then gradually die out.
Note that this seems to only happen if Vitess is waiting inside a "blocking call" (waiting for the insert to return success or error). If the insert completes, and the client connection is then closed, the transaction is immediately rolled back by Vitess as expected. For example, if the connection in the Session 1 below is closed, its transaction is immediately rolled back. If the connection in the Session 2 below is closed, while Session 1 stays open, Session 2's transaction is not immediately rolled back.
Reproduction Steps
Schema:
Vchema:
Session 1
Session 2
Expected behavior
Session 2
is immediately rolled back by Vitess.Actual behavior
Session 2
is not immediately rolled back/killed by Vitess.Session 2
is rolled back either by MySQL afterinnodb_lock_wait_timeout
, or by Vitess afterqueryserver-config-transaction-timeout
, whichever is shorter.Binary version
Git SHA:
ac3a3d69886bfacb63bffaa24e27388cc5c6f7f0
Operating system and Environment details
OS, Architecture, and any other information you can provide
about the environment.
cat /etc/os-release
):Debian GNU/Linux 9 (stretch)
uname -sr
):Linux 4.14.154-128.181.amzn2.x86_64
uname -m
):x86_64
Log Fragments
tcpdump capture showing client closing connection after its own timeout, and vitess sending error to a (half-)closed connection after mysql TX timeout:
The text was updated successfully, but these errors were encountered: