Crashing often with strange new error (Error setting MYSQL_OPTION_MULTI_STATEMENTS)

[dzadzakin]

ProxySQL version 2.0.11-124-g971c15e, codename Truls

CentOS Linux release 7.7.1908 (Core)

Hi everyone,

yesterday we started to receive multiple crashes of ProxySQL.

Inside captured information by tcpdump on ProxySQL instance there is no any multi statement query or any attempt to set it.

We also tried to disable mysql-client_multi_statements but it not helps (after saving and reloading the global variables to runtime).

+-------------------------------+----------------+
| variable_name                 | variable_value |
+-------------------------------+----------------+
| mysql-client_multi_statements | false          |
+-------------------------------+----------------+

Our Mysql clients are receiving the following errors:

Connect Error (2006) MySQL server has gone away
or
Connect Error (2002) Connection refused

Below is the ProxySQL log:

2020-05-01 21:40:08 mysql_connection.cpp:1445:handler(): [ERROR] Error setting MYSQL_OPTION_MULTI_STATEMENTS : Lost connection to MySQL server during query
2020-05-01 21:40:08 MySQL_Session.cpp:2311:handler_again___status_SETTING_MULTI_STMT(): [ERROR] Error setting multistatement on server 10.132.10.11 , 3306 : 2013, Lost connection to MySQL server during query
2020-05-01 21:40:08 MySQL_Thread.cpp:3952:run(): [ERROR] revents==POLLNVAL for FD=115, events=1, MyDSFD=115, MyConnFD=115
proxysql: MySQL_Thread.cpp:3953: void MySQL_Thread::run(): Assertion `mypolls.fds[n].revents!=0x020' failed.
Error: signal 6:
/usr/bin/proxysql(_Z13crash_handleri+0x1a)[0x4b86ba]
/lib64/libc.so.6(+0x36400)[0x7ff9d3763400]
/lib64/libc.so.6(gsignal+0x37)[0x7ff9d3763387]
/lib64/libc.so.6(abort+0x148)[0x7ff9d3764a78]
/lib64/libc.so.6(+0x2f1a6)[0x7ff9d375c1a6]
/lib64/libc.so.6(+0x2f252)[0x7ff9d375c252]
/usr/bin/proxysql(_ZN12MySQL_Thread3runEv+0x2463)[0x501873]
/usr/bin/proxysql(_Z24mysql_worker_thread_funcPv+0x6c)[0x4b19ec]
/lib64/libpthread.so.0(+0x7ea5)[0x7ff9d4943ea5]
/lib64/libc.so.6(clone+0x6d)[0x7ff9d382b8dd]
2020-05-01 21:40:09 main.cpp:1564:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2020-05-01 21:40:09 [INFO] ProxySQL version 2.0.11-124-g971c15e
2020-05-01 21:40:09 [INFO] ProxySQL SHA1 checksum: 86b9914a5c837b13e3bbb73bfc371c36c389861e
2020-05-01 21:40:09 [INFO] Angel process started ProxySQL process 9418
Standard ProxySQL MySQL Logger rev. 2.0.0714 -- MySQL_Logger.cpp -- Mon Mar 23 13:38:46 2020
Standard ProxySQL Cluster rev. 0.4.0906 -- ProxySQL_Cluster.cpp -- Mon Mar 23 13:38:46 2020
Standard ProxySQL Statistics rev. 1.4.1027 -- ProxySQL_Statistics.cpp -- Mon Mar 23 13:38:46 2020
Standard ProxySQL HTTP Server Handler rev. 1.4.1031 -- ProxySQL_HTTP_Server.cpp -- Mon Mar 23 13:38:46 2020
2020-05-01 21:40:09 ProxySQL_Admin.cpp:5458:flush_mysql_variables___database_to_runtime(): [WARNING] The default_character_set_results set to invalid value in the configuration file. Changing to default utf8
2020-05-01 21:40:09 ProxySQL_Admin.cpp:5458:flush_mysql_variables___database_to_runtime(): [WARNING] The default_CHARSET set to invalid value in the configuration file. Changing to default utf8
2020-05-01 21:40:09 ProxySQL_Admin.cpp:5458:flush_mysql_variables___database_to_runtime(): [WARNING] The default_character_set_connection set to invalid value in the configuration file. Changing to default utf8
2020-05-01 21:40:09 ProxySQL_Admin.cpp:5458:flush_mysql_variables___database_to_runtime(): [WARNING] The default_character_set_client set to invalid value in the configuration file. Changing to default utf8
2020-05-01 21:40:09 ProxySQL_Admin.cpp:5458:flush_mysql_variables___database_to_runtime(): [WARNING] The default_character_set_database set to invalid value in the configuration file. Changing to default utf8
Standard ProxySQL Admin rev. 2.0.6.0805 -- ProxySQL_Admin.cpp -- Thu Apr 30 02:35:36 2020
2020-05-01 21:40:09 [INFO] ProxySQL SHA1 checksum: 86b9914a5c837b13e3bbb73bfc371c36c389861e
Standard MySQL Threads Handler rev. 0.2.0902 -- MySQL_Thread.cpp -- Thu Apr 30 02:35:36 2020
Standard MySQL Authentication rev. 0.2.0902 -- MySQL_Authentication.cpp -- Mon Mar 23 13:38:46 2020
2020-05-01 21:40:09 [INFO] Dumping mysql_servers_incoming
+--------------+--------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname     | port | gtid_port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+--------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 10           | 10.132.10.11 | 3306 | 0         | 1000   | 0      | 1           | 2000            | 60                  | 0       | 0              |         |
| 11           | 10.132.10.12 | 3306 | 0         | 1000   | 0      | 1           | 2000            | 60                  | 0       | 0              |         |
| 11           | 10.132.10.13 | 3306 | 0         | 1000   | 0      | 1           | 2000            | 60                  | 0       | 0              |         |
+--------------+--------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
2020-05-01 21:40:09 [INFO] Dumping mysql_servers LEFT JOIN mysql_servers_incoming
+-------------+--------------+----------+------+
| mem_pointer | hostgroup_id | hostname | port |
+-------------+--------------+----------+------+
+-------------+--------------+----------+------+
2020-05-01 21:40:09 [INFO] Dumping mysql_servers JOIN mysql_servers_incoming
+--------------+--------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname     | port | gtid_port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer | gtid_port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+--------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 10           | 10.132.10.11 | 3306 | 0         | 1000   | 0      | 1           | 2000            | 60                  | 0       | 0              |         | 0           | 0         | 1000   | 0      | 1           | 2000            | 60                  | 0       | 0              |         |
| 11           | 10.132.10.12 | 3306 | 0         | 1000   | 0      | 1           | 2000            | 60                  | 0       | 0              |         | 0           | 0         | 1000   | 0      | 1           | 2000            | 60                  | 0       | 0              |         |
| 11           | 10.132.10.13 | 3306 | 0         | 1000   | 0      | 1           | 2000            | 60                  | 0       | 0              |         | 0           | 0         | 1000   | 0      | 1           | 2000            | 60                  | 0       | 0              |         |
+--------------+--------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
2020-05-01 21:40:09 [INFO] Creating new server in HG 10 : 10.132.10.11:3306 , gtid_port=0, weight=1000, status=0
2020-05-01 21:40:09 [INFO] Creating new server in HG 11 : 10.132.10.12:3306 , gtid_port=0, weight=1000, status=0
2020-05-01 21:40:09 [INFO] Creating new server in HG 11 : 10.132.10.13:3306 , gtid_port=0, weight=1000, status=0
2020-05-01 21:40:09 [INFO] New mysql_group_replication_hostgroups table
2020-05-01 21:40:09 [INFO] New mysql_galera_hostgroups table
2020-05-01 21:40:09 [INFO] Loading Galera info for (10,100,11,666,on,1,2,100,"backup_write_hostgroup acting as reader")
2020-05-01 21:40:09 [INFO] New mysql_aws_aurora_hostgroups table
2020-05-01 21:40:09 [INFO] MySQL_HostGroups_Manager::commit() locked for 2ms
Standard Query Processor rev. 2.0.6.0805 -- Query_Processor.cpp -- Thu Apr 30 02:35:36 2020
In memory Standard Query Cache (SQC) rev. 1.2.0905 -- Query_Cache.cpp -- Mon Mar 23 13:38:46 2020
2020-05-01 21:40:09 [INFO] Starting new mysql event log file /var/lib/proxysql/queries.log.00000015
Standard MySQL Monitor (StdMyMon) rev. 2.0.1226 -- MySQL_Monitor.cpp -- Mon Mar 23 13:38:46 2020
2020-05-01 21:40:09 [INFO] Latest ProxySQL version available: 2.0.11-123-g504da773

2020-05-01 21:40:09 MySQL_HostGroups_Manager.cpp:4721:update_galera_set_writer(): [WARNING] Galera: setting host 10.132.10.11:3306 as writer
2020-05-01 21:40:09 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
HID: 10 , address: 10.132.10.11 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 11 , address: 10.132.10.12 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 11 , address: 10.132.10.13 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
2020-05-01 21:40:09 [INFO] Dumping mysql_servers: ALL
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| hid | hostname     | port | gtid | weight | status | cmp | max_conns | max_lag | ssl | max_lat | comment | mem_pointer     |
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| 10  | 10.132.10.11 | 3306 | 0    | 1000   | 0      | 1   | 2000      | 60      | 0   | 0       |         | 140710916843936 |
| 11  | 10.132.10.13 | 3306 | 0    | 1000   | 0      | 1   | 2000      | 60      | 0   | 0       |         | 140710916847616 |
| 11  | 10.132.10.12 | 3306 | 0    | 1000   | 0      | 1   | 2000      | 60      | 0   | 0       |         | 140710916844096 |
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
2020-05-01 21:40:09 [INFO] Dumping mysql_servers_incoming
+--------------+--------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname     | port | gtid_port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+--------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 10           | 10.132.10.11 | 3306 | 0         | 1000   | 0      | 1           | 2000            | 60                  | 0       | 0              |         |
| 11           | 10.132.10.13 | 3306 | 0         | 1000   | 0      | 1           | 2000            | 60                  | 0       | 0              |         |
| 11           | 10.132.10.12 | 3306 | 0         | 1000   | 0      | 1           | 2000            | 60                  | 0       | 0              |         |
| 11           | 10.132.10.11 | 3306 | 0         | 1000   | 0      | 1           | 2000            | 60                  | 0       | 0              |         |
+--------------+--------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
2020-05-01 21:40:09 [INFO] Dumping mysql_servers LEFT JOIN mysql_servers_incoming
+-------------+--------------+----------+------+
| mem_pointer | hostgroup_id | hostname | port |
+-------------+--------------+----------+------+
+-------------+--------------+----------+------+
2020-05-01 21:40:09 [INFO] Dumping mysql_servers JOIN mysql_servers_incoming
+--------------+--------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname     | port | gtid_port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer | gtid_port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+--------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 11           | 10.132.10.11 | 3306 | 0         | 1000   | 0      | 1           | 2000            | 60                  | 0       | 0              |         | 0           | 0         | 1000   | 0      | 1           | 2000            | 60                  | 0       | 0              |         |
+--------------+--------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
2020-05-01 21:40:09 [INFO] Creating new server in HG 11 : 10.132.10.11:3306 , gtid_port=0, weight=1000, status=0
2020-05-01 21:40:09 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
HID: 10 , address: 10.132.10.11 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 11 , address: 10.132.10.12 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 11 , address: 10.132.10.13 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 11 , address: 10.132.10.11 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
2020-05-01 21:40:09 [INFO] Dumping mysql_servers: ALL
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| hid | hostname     | port | gtid | weight | status | cmp | max_conns | max_lag | ssl | max_lat | comment | mem_pointer     |
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| 10  | 10.132.10.11 | 3306 | 0    | 1000   | 0      | 1   | 2000      | 60      | 0   | 0       |         | 140710916843936 |
| 11  | 10.132.10.11 | 3306 | 0    | 1000   | 0      | 1   | 2000      | 60      | 0   | 0       |         | 140710839947200 |
| 11  | 10.132.10.13 | 3306 | 0    | 1000   | 0      | 1   | 2000      | 60      | 0   | 0       |         | 140710916847616 |
| 11  | 10.132.10.12 | 3306 | 0    | 1000   | 0      | 1   | 2000      | 60      | 0   | 0       |         | 140710916844096 |
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
2020-05-01 21:40:09 [INFO] MySQL_HostGroups_Manager::commit() locked for 2ms
2020-05-01 21:40:09 [INFO] Dumping current MySQL Servers structures for hostgroup 10
HID: 10 , address: 10.132.10.11 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
2020-05-01 21:40:09 [INFO] Dumping mysql_servers: HG 10
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| hid | hostname     | port | gtid | weight | status | cmp | max_conns | max_lag | ssl | max_lat | comment | mem_pointer     |
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| 10  | 10.132.10.11 | 3306 | 0    | 1000   | 0      | 1   | 2000      | 60      | 0   | 0       |         | 140710916843936 |
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
2020-05-01 21:40:09 [INFO] Dumping current MySQL Servers structures for hostgroup 100
2020-05-01 21:40:09 [INFO] Dumping mysql_servers: HG 100
+-----+----------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-------------+
| hid | hostname | port | gtid | weight | status | cmp | max_conns | max_lag | ssl | max_lat | comment | mem_pointer |
+-----+----------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-------------+
+-----+----------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-------------+
2020-05-01 21:40:09 [INFO] Dumping current MySQL Servers structures for hostgroup 11
HID: 11 , address: 10.132.10.12 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 11 , address: 10.132.10.13 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 11 , address: 10.132.10.11 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
2020-05-01 21:40:09 [INFO] Dumping mysql_servers: HG 11
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| hid | hostname     | port | gtid | weight | status | cmp | max_conns | max_lag | ssl | max_lat | comment | mem_pointer     |
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| 11  | 10.132.10.11 | 3306 | 0    | 1000   | 0      | 1   | 2000      | 60      | 0   | 0       |         | 140710839947200 |
| 11  | 10.132.10.12 | 3306 | 0    | 1000   | 0      | 1   | 2000      | 60      | 0   | 0       |         | 140710916844096 |
| 11  | 10.132.10.13 | 3306 | 0    | 1000   | 0      | 1   | 2000      | 60      | 0   | 0       |         | 140710916847616 |
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
2020-05-01 21:40:09 [INFO] Dumping current MySQL Servers structures for hostgroup 666
2020-05-01 21:40:09 [INFO] Dumping mysql_servers: HG 666
+-----+----------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-------------+
| hid | hostname | port | gtid | weight | status | cmp | max_conns | max_lag | ssl | max_lat | comment | mem_pointer |
+-----+----------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-------------+
+-----+----------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-------------+
2020-05-01 21:40:09 MySQL_HostGroups_Manager.cpp:4721:update_galera_set_writer(): [WARNING] Galera: setting host 10.132.10.12:3306 as writer
2020-05-01 21:40:09 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
HID: 10 , address: 10.132.10.11 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 11 , address: 10.132.10.12 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 11 , address: 10.132.10.13 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 11 , address: 10.132.10.11 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
2020-05-01 21:40:09 [INFO] Dumping mysql_servers: ALL
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| hid | hostname     | port | gtid | weight | status | cmp | max_conns | max_lag | ssl | max_lat | comment | mem_pointer     |
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| 10  | 10.132.10.11 | 3306 | 0    | 1000   | 0      | 1   | 2000      | 60      | 0   | 0       |         | 140710916843936 |
| 11  | 10.132.10.11 | 3306 | 0    | 1000   | 0      | 1   | 2000      | 60      | 0   | 0       |         | 140710839947200 |
| 11  | 10.132.10.13 | 3306 | 0    | 1000   | 0      | 1   | 2000      | 60      | 0   | 0       |         | 140710916847616 |
| 11  | 10.132.10.12 | 3306 | 0    | 1000   | 0      | 1   | 2000      | 60      | 0   | 0       |         | 140710916844096 |
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
2020-05-01 21:40:09 [INFO] Dumping mysql_servers_incoming
+--------------+--------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname     | port | gtid_port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+--------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 100          | 10.132.10.11 | 3306 | 0         | 1000   | 0      | 1           | 2000            | 60                  | 0       | 0              |         |
| 10           | 10.132.10.12 | 3306 | 0         | 1000   | 0      | 1           | 2000            | 60                  | 0       | 0              |         |
| 11           | 10.132.10.11 | 3306 | 0         | 1000   | 0      | 1           | 2000            | 60                  | 0       | 0              |         |
+--------------+--------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
2020-05-01 21:40:09 [INFO] Dumping mysql_servers LEFT JOIN mysql_servers_incoming
+-----------------+--------------+--------------+------+
| mem_pointer     | hostgroup_id | hostname     | port |
+-----------------+--------------+--------------+------+
| 140710916843936 | 10           | 10.132.10.11 | 3306 |
| 140710916847616 | 11           | 10.132.10.13 | 3306 |
| 140710916844096 | 11           | 10.132.10.12 | 3306 |
+-----------------+--------------+--------------+------+
2020-05-01 21:40:09 MySQL_HostGroups_Manager.cpp:1268:commit(): [WARNING] Removed server at address 140710916843936, hostgroup 10, address 10.132.10.11 port 3306. Setting status OFFLINE HARD and immediately dropping all free connections. Used connections will be dropped when trying to use them
2020-05-01 21:40:09 MySQL_HostGroups_Manager.cpp:1268:commit(): [WARNING] Removed server at address 140710916847616, hostgroup 11, address 10.132.10.13 port 3306. Setting status OFFLINE HARD and immediately dropping all free connections. Used connections will be dropped when trying to use them
2020-05-01 21:40:09 MySQL_HostGroups_Manager.cpp:1268:commit(): [WARNING] Removed server at address 140710916844096, hostgroup 11, address 10.132.10.12 port 3306. Setting status OFFLINE HARD and immediately dropping all free connections. Used connections will be dropped when trying to use them
2020-05-01 21:40:09 [INFO] Dumping mysql_servers JOIN mysql_servers_incoming
+--------------+--------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname     | port | gtid_port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer | gtid_port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+--------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 100          | 10.132.10.11 | 3306 | 0         | 1000   | 0      | 1           | 2000            | 60                  | 0       | 0              |         | 0           | 0         | 1000   | 0      | 1           | 2000            | 60                  | 0       | 0              |         |
| 10           | 10.132.10.12 | 3306 | 0         | 1000   | 0      | 1           | 2000            | 60                  | 0       | 0              |         | 0           | 0         | 1000   | 0      | 1           | 2000            | 60                  | 0       | 0              |         |
+--------------+--------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
2020-05-01 21:40:09 [INFO] Creating new server in HG 100 : 10.132.10.11:3306 , gtid_port=0, weight=1000, status=0
2020-05-01 21:40:09 [INFO] Creating new server in HG 10 : 10.132.10.12:3306 , gtid_port=0, weight=1000, status=0
2020-05-01 21:40:09 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
HID: 10 , address: 10.132.10.11 , port: 3306 , gtid_port: 0 , weight: 1000 , status: OFFLINE_HARD , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 10 , address: 10.132.10.12 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 11 , address: 10.132.10.12 , port: 3306 , gtid_port: 0 , weight: 1000 , status: OFFLINE_HARD , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 11 , address: 10.132.10.13 , port: 3306 , gtid_port: 0 , weight: 1000 , status: OFFLINE_HARD , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 11 , address: 10.132.10.11 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 100 , address: 10.132.10.11 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
2020-05-01 21:40:09 [INFO] Dumping mysql_servers: ALL
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| hid | hostname     | port | gtid | weight | status | cmp | max_conns | max_lag | ssl | max_lat | comment | mem_pointer     |
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| 10  | 10.132.10.11 | 3306 | 0    | 1000   | 3      | 1   | 2000      | 60      | 0   | 0       |         | 140710916843936 |
| 100 | 10.132.10.11 | 3306 | 0    | 1000   | 0      | 1   | 2000      | 60      | 0   | 0       |         | 140710846382016 |
| 11  | 10.132.10.11 | 3306 | 0    | 1000   | 0      | 1   | 2000      | 60      | 0   | 0       |         | 140710839947200 |
| 11  | 10.132.10.13 | 3306 | 0    | 1000   | 3      | 1   | 2000      | 60      | 0   | 0       |         | 140710916847616 |
| 11  | 10.132.10.12 | 3306 | 0    | 1000   | 3      | 1   | 2000      | 60      | 0   | 0       |         | 140710916844096 |
| 10  | 10.132.10.12 | 3306 | 0    | 1000   | 0      | 1   | 2000      | 60      | 0   | 0       |         | 140710846382176 |
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
2020-05-01 21:40:09 [INFO] MySQL_HostGroups_Manager::commit() locked for 2ms
2020-05-01 21:40:09 [INFO] Dumping current MySQL Servers structures for hostgroup 10
HID: 10 , address: 10.132.10.11 , port: 3306 , gtid_port: 0 , weight: 1000 , status: OFFLINE_HARD , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 10 , address: 10.132.10.12 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
2020-05-01 21:40:09 [INFO] Dumping mysql_servers: HG 10
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| hid | hostname     | port | gtid | weight | status | cmp | max_conns | max_lag | ssl | max_lat | comment | mem_pointer     |
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| 10  | 10.132.10.11 | 3306 | 0    | 1000   | 3      | 1   | 2000      | 60      | 0   | 0       |         | 140710916843936 |
| 10  | 10.132.10.12 | 3306 | 0    | 1000   | 0      | 1   | 2000      | 60      | 0   | 0       |         | 140710846382176 |
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
2020-05-01 21:40:09 [INFO] Dumping current MySQL Servers structures for hostgroup 100
HID: 100 , address: 10.132.10.11 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
2020-05-01 21:40:09 [INFO] Dumping mysql_servers: HG 100
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| hid | hostname     | port | gtid | weight | status | cmp | max_conns | max_lag | ssl | max_lat | comment | mem_pointer     |
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| 100 | 10.132.10.11 | 3306 | 0    | 1000   | 0      | 1   | 2000      | 60      | 0   | 0       |         | 140710846382016 |
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
2020-05-01 21:40:09 [INFO] Dumping current MySQL Servers structures for hostgroup 11
HID: 11 , address: 10.132.10.12 , port: 3306 , gtid_port: 0 , weight: 1000 , status: OFFLINE_HARD , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 11 , address: 10.132.10.13 , port: 3306 , gtid_port: 0 , weight: 1000 , status: OFFLINE_HARD , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 11 , address: 10.132.10.11 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
2020-05-01 21:40:09 [INFO] Dumping mysql_servers: HG 11
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| hid | hostname     | port | gtid | weight | status | cmp | max_conns | max_lag | ssl | max_lat | comment | mem_pointer     |
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| 11  | 10.132.10.11 | 3306 | 0    | 1000   | 0      | 1   | 2000      | 60      | 0   | 0       |         | 140710839947200 |
| 11  | 10.132.10.12 | 3306 | 0    | 1000   | 3      | 1   | 2000      | 60      | 0   | 0       |         | 140710916844096 |
| 11  | 10.132.10.13 | 3306 | 0    | 1000   | 3      | 1   | 2000      | 60      | 0   | 0       |         | 140710916847616 |
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
2020-05-01 21:40:09 [INFO] Dumping current MySQL Servers structures for hostgroup 666
2020-05-01 21:40:09 [INFO] Dumping mysql_servers: HG 666
+-----+----------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-------------+
| hid | hostname | port | gtid | weight | status | cmp | max_conns | max_lag | ssl | max_lat | comment | mem_pointer |
+-----+----------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-------------+
+-----+----------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-------------+
2020-05-01 21:40:09 MySQL_HostGroups_Manager.cpp:4721:update_galera_set_writer(): [WARNING] Galera: setting host 10.132.10.13:3306 as writer
2020-05-01 21:40:09 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
HID: 10 , address: 10.132.10.12 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 11 , address: 10.132.10.11 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 100 , address: 10.132.10.11 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
2020-05-01 21:40:09 [INFO] Dumping mysql_servers: ALL
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| hid | hostname     | port | gtid | weight | status | cmp | max_conns | max_lag | ssl | max_lat | comment | mem_pointer     |
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| 10  | 10.132.10.12 | 3306 | 0    | 1000   | 0      | 1   | 2000      | 60      | 0   | 0       |         | 140710846382176 |
| 100 | 10.132.10.11 | 3306 | 0    | 1000   | 0      | 1   | 2000      | 60      | 0   | 0       |         | 140710846382016 |
| 11  | 10.132.10.11 | 3306 | 0    | 1000   | 0      | 1   | 2000      | 60      | 0   | 0       |         | 140710839947200 |
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
2020-05-01 21:40:09 [INFO] Dumping mysql_servers_incoming
+--------------+--------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname     | port | gtid_port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+--------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 10           | 10.132.10.11 | 3306 | 0         | 1000   | 3      | 1           | 2000            | 60                  | 0       | 0              |         |
| 100          | 10.132.10.12 | 3306 | 0         | 1000   | 0      | 1           | 2000            | 60                  | 0       | 0              |         |
| 100          | 10.132.10.11 | 3306 | 0         | 1000   | 0      | 1           | 2000            | 60                  | 0       | 0              |         |
| 10           | 10.132.10.13 | 3306 | 0         | 1000   | 0      | 1           | 2000            | 60                  | 0       | 0              |         |
| 11           | 10.132.10.11 | 3306 | 0         | 1000   | 0      | 1           | 2000            | 60                  | 0       | 0              |         |
| 11           | 10.132.10.12 | 3306 | 0         | 1000   | 0      | 1           | 2000            | 60                  | 0       | 0              |         |
+--------------+--------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
2020-05-01 21:40:09 [INFO] Dumping mysql_servers LEFT JOIN mysql_servers_incoming
+-----------------+--------------+--------------+------+
| mem_pointer     | hostgroup_id | hostname     | port |
+-----------------+--------------+--------------+------+
| 140710846382176 | 10           | 10.132.10.12 | 3306 |
+-----------------+--------------+--------------+------+
2020-05-01 21:40:09 MySQL_HostGroups_Manager.cpp:1268:commit(): [WARNING] Removed server at address 140710846382176, hostgroup 10, address 10.132.10.12 port 3306. Setting status OFFLINE HARD and immediately dropping all free connections. Used connections will be dropped when trying to use them
2020-05-01 21:40:09 [INFO] Dumping mysql_servers JOIN mysql_servers_incoming
+--------------+--------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname     | port | gtid_port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer | gtid_port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+--------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 10           | 10.132.10.11 | 3306 | 0         | 1000   | 3      | 1           | 2000            | 60                  | 0       | 0              |         | 0           | 0         | 1000   | 3      | 1           | 2000            | 60                  | 0       | 0              |         |
| 100          | 10.132.10.12 | 3306 | 0         | 1000   | 0      | 1           | 2000            | 60                  | 0       | 0              |         | 0           | 0         | 1000   | 0      | 1           | 2000            | 60                  | 0       | 0              |         |
| 10           | 10.132.10.13 | 3306 | 0         | 1000   | 0      | 1           | 2000            | 60                  | 0       | 0              |         | 0           | 0         | 1000   | 0      | 1           | 2000            | 60                  | 0       | 0              |         |
| 11           | 10.132.10.12 | 3306 | 0         | 1000   | 0      | 1           | 2000            | 60                  | 0       | 0              |         | 0           | 0         | 1000   | 0      | 1           | 2000            | 60                  | 0       | 0              |         |
+--------------+--------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
2020-05-01 21:40:09 [INFO] Creating new server in HG 10 : 10.132.10.11:3306 , gtid_port=0, weight=1000, status=3
2020-05-01 21:40:09 [INFO] Creating new server in HG 100 : 10.132.10.12:3306 , gtid_port=0, weight=1000, status=0
2020-05-01 21:40:09 [INFO] Creating new server in HG 10 : 10.132.10.13:3306 , gtid_port=0, weight=1000, status=0
2020-05-01 21:40:09 [INFO] Creating new server in HG 11 : 10.132.10.12:3306 , gtid_port=0, weight=1000, status=0
2020-05-01 21:40:09 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
HID: 10 , address: 10.132.10.12 , port: 3306 , gtid_port: 0 , weight: 1000 , status: OFFLINE_HARD , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 10 , address: 10.132.10.11 , port: 3306 , gtid_port: 0 , weight: 1000 , status: OFFLINE_HARD , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 10 , address: 10.132.10.13 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 11 , address: 10.132.10.11 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 11 , address: 10.132.10.12 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 100 , address: 10.132.10.11 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 100 , address: 10.132.10.12 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
2020-05-01 21:40:09 [INFO] Dumping mysql_servers: ALL
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| hid | hostname     | port | gtid | weight | status | cmp | max_conns | max_lag | ssl | max_lat | comment | mem_pointer     |
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| 10  | 10.132.10.12 | 3306 | 0    | 1000   | 3      | 1   | 2000      | 60      | 0   | 0       |         | 140710846382176 |
| 100 | 10.132.10.12 | 3306 | 0    | 1000   | 0      | 1   | 2000      | 60      | 0   | 0       |         | 140710916844096 |
| 100 | 10.132.10.11 | 3306 | 0    | 1000   | 0      | 1   | 2000      | 60      | 0   | 0       |         | 140710846382016 |
| 11  | 10.132.10.12 | 3306 | 0    | 1000   | 0      | 1   | 2000      | 60      | 0   | 0       |         | 140710839947680 |
| 11  | 10.132.10.11 | 3306 | 0    | 1000   | 0      | 1   | 2000      | 60      | 0   | 0       |         | 140710839947200 |
| 10  | 10.132.10.13 | 3306 | 0    | 1000   | 0      | 1   | 2000      | 60      | 0   | 0       |         | 140710916843936 |
| 10  | 10.132.10.11 | 3306 | 0    | 1000   | 3      | 1   | 2000      | 60      | 0   | 0       |         | 140710916847616 |
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
2020-05-01 21:40:09 [INFO] MySQL_HostGroups_Manager::commit() locked for 2ms
2020-05-01 21:40:09 [INFO] Dumping current MySQL Servers structures for hostgroup 10
HID: 10 , address: 10.132.10.12 , port: 3306 , gtid_port: 0 , weight: 1000 , status: OFFLINE_HARD , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 10 , address: 10.132.10.11 , port: 3306 , gtid_port: 0 , weight: 1000 , status: OFFLINE_HARD , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 10 , address: 10.132.10.13 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
2020-05-01 21:40:09 [INFO] Dumping mysql_servers: HG 10
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| hid | hostname     | port | gtid | weight | status | cmp | max_conns | max_lag | ssl | max_lat | comment | mem_pointer     |
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| 10  | 10.132.10.11 | 3306 | 0    | 1000   | 3      | 1   | 2000      | 60      | 0   | 0       |         | 140710916847616 |
| 10  | 10.132.10.12 | 3306 | 0    | 1000   | 3      | 1   | 2000      | 60      | 0   | 0       |         | 140710846382176 |
| 10  | 10.132.10.13 | 3306 | 0    | 1000   | 0      | 1   | 2000      | 60      | 0   | 0       |         | 140710916843936 |
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
2020-05-01 21:40:09 [INFO] Dumping current MySQL Servers structures for hostgroup 100
HID: 100 , address: 10.132.10.11 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 100 , address: 10.132.10.12 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
2020-05-01 21:40:09 [INFO] Dumping mysql_servers: HG 100
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| hid | hostname     | port | gtid | weight | status | cmp | max_conns | max_lag | ssl | max_lat | comment | mem_pointer     |
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| 100 | 10.132.10.11 | 3306 | 0    | 1000   | 0      | 1   | 2000      | 60      | 0   | 0       |         | 140710846382016 |
| 100 | 10.132.10.12 | 3306 | 0    | 1000   | 0      | 1   | 2000      | 60      | 0   | 0       |         | 140710916844096 |
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
2020-05-01 21:40:09 [INFO] Dumping current MySQL Servers structures for hostgroup 11
HID: 11 , address: 10.132.10.11 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 11 , address: 10.132.10.12 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 2000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment: 
2020-05-01 21:40:09 [INFO] Dumping mysql_servers: HG 11
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| hid | hostname     | port | gtid | weight | status | cmp | max_conns | max_lag | ssl | max_lat | comment | mem_pointer     |
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| 11  | 10.132.10.11 | 3306 | 0    | 1000   | 0      | 1   | 2000      | 60      | 0   | 0       |         | 140710839947200 |
| 11  | 10.132.10.12 | 3306 | 0    | 1000   | 0      | 1   | 2000      | 60      | 0   | 0       |         | 140710839947680 |
+-----+--------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
2020-05-01 21:40:09 [INFO] Dumping current MySQL Servers structures for hostgroup 666
2020-05-01 21:40:09 [INFO] Dumping mysql_servers: HG 666
+-----+----------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-------------+
| hid | hostname | port | gtid | weight | status | cmp | max_conns | max_lag | ssl | max_lat | comment | mem_pointer |
+-----+----------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-------------+
+-----+----------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-------------+
2020-05-01 21:44:09 MySQL_Session.cpp:1454:handler_again___status_PINGING_SERVER(): [ERROR] Detected a broken connection during ping on (11,10.132.10.11,3306) , FD (Conn:70 , MyDS:70) : 2013, Lost connection to MySQL server during query

Below is the output of core dump with gdb:

# gdb /usr/bin/proxysql /var/lib/proxysql/core.9919
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-119.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/proxysql...done.
[New LWP 9929]
[New LWP 9935]
[New LWP 9930]
[New LWP 9932]
[New LWP 9934]
[New LWP 9941]
[New LWP 9936]
[New LWP 9942]
[New LWP 9939]
[New LWP 9940]
[New LWP 9943]
[New LWP 9944]
[New LWP 9947]
[New LWP 10085]
[New LWP 10343]
[New LWP 10344]
[New LWP 10348]
[New LWP 9919]
[New LWP 9920]
[New LWP 9922]
[New LWP 9923]
[New LWP 9924]
[New LWP 9945]
[New LWP 9925]
[New LWP 10345]
[New LWP 10346]
[New LWP 9926]
[New LWP 10347]
[New LWP 9928]
[New LWP 9931]
[New LWP 9921]
[New LWP 9946]
[New LWP 9927]
[New LWP 9933]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/bin/proxysql --idle-threads -c /etc/proxysql.cnf'.
Program terminated with signal 6, Aborted.
#0  0x00007ff9d3763387 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install proxysql-2.0.11-1.x86_64
(gdb) 

Can you help please?

Thanks

[renecannao]

Hi.
Thank you for the report.
We will work in identifying the bug and fix it.

Meanwhile I would like to find a way to reduce the probability of hitting such bug.
Do you have a lot of errors like 2013, Lost connection to MySQL server during ... in the error log?
If will be very useful if you could share the full error log.
Thanks

[dzadzakin]

Hi @renecannao,

and thanks for your quick reply.

Yes, we have a lot of such errors (2013, Lost connection to MySQL server during), after a lot of reading I realised that in certain situations they can be considered "normal" and not creating the problems for the app/cluster itself.

After enabled general log on each mysql node I found this strange activity by monitor user (the time is corresponding to the crash):

2020-05-02T13:25:02.711122+01:00    756720 Connect  [email protected] on  using TCP/IP
2020-05-02T13:25:02.711571+01:00    756720 Query    SELECT (SELECT VARIABLE_VALUE FROM performance_schema.global_status WHERE VARIABLE_NAME='WSREP_LOCAL_STATE') wsrep_local_state, @@read_only read_only, (SELECT VARIABLE_VALUE FROM performance_schema.global_status WHERE VARIABLE_NAME='WSREP_LOCAL_RECV_QUEUE') wsrep_local_recv_queue , @@wsrep_desync wsrep_desync, @@wsrep_reject_queries wsrep_reject_queries, @@wsrep_sst_donor_rejects_queries wsrep_sst_donor_rejects_queries, (SELECT VARIABLE_VALUE FROM performance_schema.global_status WHERE VARIABLE_NAME='WSREP_CLUSTER_STATUS') wsrep_cluster_status , (SELECT COALESCE(MAX(VARIABLE_VALUE),'DISABLED') FROM performance_schema.global_variables WHERE variable_name='pxc_maint_mode') pxc_maint_mode
2020-05-02T13:25:02.719553+01:00    756720 Query    SET wait_timeout=20
2020-05-02T13:25:02.726156+01:00    756720 Query    SHOW SLAVE STATUS
2020-05-02T13:25:02.735094+01:00    756721 Connect  [email protected] on  using TCP/IP
2020-05-02T13:25:02.735397+01:00    756721 Quit 

Someone have an idea why Monitor should execute this query SET wait_timeout=20 and how to avoid it?

The complete proxysql.log was sent to renecannao gmail.

Thanks a lot,
Let's hope to find the reason and mitigate it asap

[dzadzakin]

Hi again,

the post below (about SET wait_timeout=20 was unrelated).

The crash continues even after setting and loading the mysql variables to runtime:

update global_variables set variable_value = 'false' where variable_name = 'mysql-monitor_wait_timeout';

Even after disabling ProxySQL Cache it's still crashing...

If someone has an idea how to decrease the chances to hit this bug please share.

Thanks

[renecannao]

Hi @dzadzakin .
Thank you for the log.
I haven't review it yet.

Ignore what monitor user is doing, it is absolutely not related.

What I would recommend now (before checking the log) is to increase global wait_timeout on MySQL server (not in proxysql).

Thanks

[dzadzakin]

Hi @renecannao, Thanks for your reply. Following your recommendation I discovered a strange wait_timeout setting on each node of our XtraDB Percona Cluster: mysql> SELECT @@global.wait_timeout , @@session.wait_timeout; +-----------------------+------------------------+ | @@global.wait_timeout | @@session.wait_timeout | +-----------------------+------------------------+ | 150 | 28800 | +-----------------------+------------------------+ 1 row in set (0.00 sec) I always used this query (which probably returns only the value for current session): mysql> show variables like '%wait_timeout%'; +-------------------------------------+----------+ | Variable_name | Value | +-------------------------------------+----------+ | innodb_lock_wait_timeout | 120 | | innodb_print_lock_wait_timeout_info | OFF | | lock_wait_timeout | 31536000 | | wait_timeout | 28800 | +-------------------------------------+----------+ 4 rows in set (0.00 sec) I set global wait_timeout to 28800 too: mysql> SET GLOBAL wait_timeout = 28800; Query OK, 0 rows affected (0.00 sec) mysql> SELECT @@global.wait_timeout , @@session.wait_timeout; +-----------------------+------------------------+ | @@global.wait_timeout | @@session.wait_timeout | +-----------------------+------------------------+ | 28800 | 28800 | +-----------------------+------------------------+ 1 row in set (0.00 sec) Can you please confirm that this could be a good reason of multiple crashes (almost 50 in 2 days)? I'll continue to monitor the situation, it's about 45 min since this change and and no crash. Thanks a lot

…

On Sun, 3 May 2020 at 14:02, René Cannaò ***@***.***> wrote: Hi @dzadzakin <https://github.com/dzadzakin> . Thank you for the log. I haven't review it yet. Ignore what monitor user is doing, it is absolutely not related. What I would recommend now (before checking the log) is to increase global wait_timeout on MySQL server (not in proxysql). Thanks — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2727 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AESFGSKJCTL5ZUBSUYD7CEDRPVML5ANCNFSM4MXOIDVQ> .

[dzadzakin]

Hi all,

low wait_timeout (150) caused the crash and plenty of errors related to pings.
Setting it to default value (28800) solved the issue and all errors are gone.

Thanks all,
I'll close this issue.

[renecannao]

Hi @dzadzakin .
I am glad the workaround works for you.
Although, it is a workaround, and now a solution.
I am opening this issue again because there is a bug we need to fix.

Thanks

[dzadzakin]

Hi @renecannao,

and thanks for the workaround and the fix.

Keep up the great work with ProxySQL,
Best regards

dzadzakin

[renecannao]

Thank you for the feedback @dzadzakin .

Closing, as fixed in 2.0.12