Connection liveness/goodeness check and AWS Aurora

[oakad]

I had an application running against AWS Aurora (3.05.2 - the most recent version) when out of sudden my application started returning a large number of "unknown prepared statement handler" errors.

Error 1243 (HY000): Unknown prepared statement handler (14) given to mysql_stmt_precheck

Upon investigation I figured out that Aurora node had crashed and recovered (the server log had a lot of chatter around the time issue had appeared, with all the usual reboot and "recovering after a crash" entries).

However, Go application could not recover: it kept getting the 1234 errors as if connection was considered good all along despite the server crash and restart.

CheckConnLiveness was left at default "true".

I have contemplated setting a SetConnMaxLifetime, but I have a large number of prepared statements so an overhead of creating a new connection is quite high. I would rather keep my connections long lived.

What will be really helpful is an option to close the connection on any sql error instead of returning it to the connection pool.

Is it possible to do so in the current driver version? Will it be feasible to add such a feature: "discard connection on any/some SQL errors"?

[oakad]

We have further investigated the Aurora issue and apparently, this is not a bug, but a side effect of a feature called "zero downtime patching" used by AWS to apply minor changes to their database instances. Nevertheless, it is a problem breaking any application relying on prepared statements and possibly other session state, such as temporary tables or session variables.

It also appears that adding a workaround for the issue is not difficult; (*mysqlConn).markBadConn can be modified to check a user configurable list of MySQL error codes instead of only checking for errBadConnNoWrite. For me, an additional check for error 1234 will solve the issue (and there may be other errors that warrant a compulsory reconnect).