VTGate does not rollback in-flight transactions when shut down

[mikrobi]

We experience a bunch of

vttablet: rpc error: code = ResourceExhausted desc = transaction pool connection limit exceeded

errors each time we deploy VTGate. After discussing this problem in Slack @sougou noted that VTGate does not rollback in-flight transactions on shut down

To avoid leaving transactions unfinished in the tablet where they would linger until the transactions timeout, we probably want them to rollback once the lame duck period has expired?

/cc @drogart and @tomkrouper
/cc @dweitzman who might also be interested in this issue

[dweitzman]

As a note: this is specifically with the mysql protocol where a transaction happens on a single gate. It wouldn't be an issue with grpc, where gates are stateless and interchangeable (although the clients of the gates might fail to kill grpc transactions if those clients shut down ungracefully).

[drogart]

Very interesting!

So, the hypothesis is that in flight txns on a vtgate do not complete during the lame duck period, then get left hanging on the tablet and consume a connection pool slot?

I guess we can either:

• bring the mysql protocol behavior in line with the grpc behavior
• increase the lame duck period (though this will still leave a long tail of txns this could happen to)

Any other ideas to alleviate this?

[dweitzman]

The useful behavior here would be for vtgate to roll back any in-progress mysql protocol transactions at the end of the lame duck period.