Stalled MySQL thread leading to ProxySQL lock up.

[bmildren]

ProxySQL: v1.4.10-1-g5eb0f3e
OS: Ubuntu 14.04

We had a couple of MySQL hosts that were experiencing intermittent soft lockups due to a build issue. Initially this led to the watchdog forcing an assertion loop across all of our ProxySQL nodes.

We updated the config to add restart_on_missing_heartbeats=0, however this led to a lock up on all of ProxySQL nodes when the problematic hosts were restarted, which only resolved when the hosts came back up again 10m later.

Ideally, to avoid affecting all connections, the stalled MySQL thread should just lead to something along the lines of a connect timeout and subsequent shunning of the node.

I can supply the logs directly 👍

[renecannao]

Hi Ben,

the watchdog forcing an assertion is a mechanism to unblock proxysql (the hard way, with a restart) in the event it stalls for unknown reasons.
The fact that the reason of the stall is unknown, it needs further debugging with a core dump.

We are continuing this conversation privately.

[renecannao]

@bmildren : is it ok to close this?

[renecannao]

Closing