Proxysql 2.0.17 | Crash (proxysql: MySQL_Protocol.cpp:711: bool MySQL_Protocol)

[aakashmuthuramalingam]

Hi,

Proxysql is getting crashed continuously whenever the application is connected. But the same behavior does not happen when doing heavy workload on sysbench, it is working perfectly with it. Also, the application is just creating less than of 10 connection and the proxysql is getting crashed immediately with the below error.

proxysql: MySQL_Protocol.cpp:711: bool MySQL_Protocol::generate_pkt_OK(bool, void**, unsigned int*, uint8_t, unsigned int, uint64_t, uint16_t, uint16_t, char*): Assertion `0' failed

Error: signal 6:
/usr/bin/proxysql(_Z13crash_handleri+0x1a)[0x4d5ffa]
/lib64/libc.so.6(+0x36400)[0x7fefbc111400]
/lib64/libc.so.6(gsignal+0x37)[0x7fefbc111387]
/lib64/libc.so.6(abort+0x148)[0x7fefbc112a78]
/lib64/libc.so.6(+0x2f1a6)[0x7fefbc10a1a6]
/lib64/libc.so.6(+0x2f252)[0x7fefbc10a252]
/usr/bin/proxysql(_ZN14MySQL_Protocol15generate_pkt_OKEbPPvPjhjmttPc+0x540)[0x53ee80]
/usr/bin/proxysql(_ZN13MySQL_Session26MySQL_Result_to_MySQL_wireEP8st_mysqlP15MySQL_ResultSetP17MySQL_Data_Stream+0x31f)[0x52605f]
/usr/bin/proxysql(_ZN13MySQL_Session7handlerEv+0x2600)[0x538000]
/usr/bin/proxysql(_ZN12MySQL_Thread20process_all_sessionsEv+0x3bc)[0x516bac]
/usr/bin/proxysql(_ZN12MySQL_Thread3runEv+0xc4c)[0x51db5c]
/usr/bin/proxysql(_Z24mysql_worker_thread_funcPv+0x6c)[0x4ceefc]
/lib64/libpthread.so.0(+0x7ea5)[0x7fefbd2f1ea5]
/lib64/libc.so.6(clone+0x6d)[0x7fefbc1d98dd]
2021-04-15 03:24:40 main.cpp:1574:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2021-04-15 03:24:40 [INFO] ProxySQL version 2.0.17-16-gf6d9be1
2021-04-15 03:24:40 [INFO] ProxySQL SHA1 checksum: ab2546523db645833192f38249d9f7df2beb8f78
2021-04-15 03:24:40 [INFO] Angel process started ProxySQL process 154470
Standard ProxySQL MySQL Logger rev. 2.0.0714 -- MySQL_Logger.cpp -- Wed Jan 20 14:30:00 2021

This behavior is repeating immediately once connected with application, so now in our production we have removed proxysql for a while until it gets fixed.

OS: CentOS 7
Proxysql Version: proxysql --version
ProxySQL version 2.0.17-16-gf6d9be1, codename Truls
CPU: 16
Memory: 30GB
Backend DB Server: MariaDB 10.4 Galera Cluster

The proxysql downloaded from the site proxysql.com. The below is the complete error log before and after crash

2021-04-15 03:22:40 [INFO] Dumping mysql_servers: HG 3
+-----+------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| hid | hostname | port | gtid | weight | status | cmp | max_conns | max_lag | ssl | max_lat | comment | mem_pointer |
+-----+------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| 3 | 10.1.11.59 | 3306 | 0 | 1 | 0 | 0 | 4000 | 0 | 0 | 0 | | 140667308408992 |
| 3 | 10.6.4.3 | 3306 | 0 | 1 | 0 | 0 | 4000 | 0 | 0 | 0 | | 140667305787552 |
+-----+------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
2021-04-15 03:22:40 [INFO] Dumping current MySQL Servers structures for hostgroup 4
2021-04-15 03:22:40 [INFO] Dumping mysql_servers: HG 4
+-----+----------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-------------+
| hid | hostname | port | gtid | weight | status | cmp | max_conns | max_lag | ssl | max_lat | comment | mem_pointer |
+-----+----------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-------------+
+-----+----------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-------------+
2021-04-15 03:22:57 [INFO] Killing query 8
2021-04-15 03:22:58 mysql_connection.cpp:1477:handler(): [ERROR] Failed SET NAMES: Commands out of sync; you can't run this command now
2021-04-15 03:22:58 MySQL_Session.cpp:2232:handler_again___status_CHANGING_CHARSET(): [ERROR] Detected a broken connection during SET NAMES on 10.6.4.3 , 3306 : 2014, Commands out o
f sync; you can't run this command now
2021-04-15 03:23:10 mysql_connection.cpp:1477:handler(): [ERROR] Failed SET NAMES: Commands out of sync; you can't run this command now
2021-04-15 03:23:10 MySQL_Session.cpp:2232:handler_again___status_CHANGING_CHARSET(): [ERROR] Detected a broken connection during SET NAMES on 10.6.4.3 , 3306 : 2014, Commands out o
f sync; you can't run this command now
2021-04-15 03:24:07 [INFO] Killing query 44
proxysql: MySQL_Protocol.cpp:711: bool MySQL_Protocol::generate_pkt_OK(bool, void**, unsigned int*, uint8_t, unsigned int, uint64_t, uint16_t, uint16_t, char*): Assertion `0' failed
.
Error: signal 6:
/usr/bin/proxysql(_Z13crash_handleri+0x1a)[0x4d5ffa]
/lib64/libc.so.6(+0x36400)[0x7fefbc111400]
/lib64/libc.so.6(gsignal+0x37)[0x7fefbc111387]
/lib64/libc.so.6(abort+0x148)[0x7fefbc112a78]
/lib64/libc.so.6(+0x2f1a6)[0x7fefbc10a1a6]
/lib64/libc.so.6(+0x2f252)[0x7fefbc10a252]
/lib64/libc.so.6(gsignal+0x37)[0x7fefbc111387]
/lib64/libc.so.6(abort+0x148)[0x7fefbc112a78]
/lib64/libc.so.6(+0x2f1a6)[0x7fefbc10a1a6]
/lib64/libc.so.6(+0x2f252)[0x7fefbc10a252]
/usr/bin/proxysql(_ZN14MySQL_Protocol15generate_pkt_OKEbPPvPjhjmttPc+0x540)[0x53ee80]
/usr/bin/proxysql(_ZN13MySQL_Session26MySQL_Result_to_MySQL_wireEP8st_mysqlP15MySQL_ResultSetP17MySQL_Data_Stream+0x31f)[0x52605f]
/usr/bin/proxysql(_ZN13MySQL_Session7handlerEv+0x2600)[0x538000]
/usr/bin/proxysql(_ZN12MySQL_Thread20process_all_sessionsEv+0x3bc)[0x516bac]
/usr/bin/proxysql(_ZN12MySQL_Thread3runEv+0xc4c)[0x51db5c]
/usr/bin/proxysql(_Z24mysql_worker_thread_funcPv+0x6c)[0x4ceefc]
/lib64/libpthread.so.0(+0x7ea5)[0x7fefbd2f1ea5]
/lib64/libc.so.6(clone+0x6d)[0x7fefbc1d98dd]
2021-04-15 03:24:40 main.cpp:1574:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2021-04-15 03:24:40 [INFO] ProxySQL version 2.0.17-16-gf6d9be1
2021-04-15 03:24:40 [INFO] ProxySQL SHA1 checksum: ab2546523db645833192f38249d9f7df2beb8f78
2021-04-15 03:24:40 [INFO] Angel process started ProxySQL process 154470
Standard ProxySQL MySQL Logger rev. 2.0.0714 -- MySQL_Logger.cpp -- Wed Jan 20 14:30:00 2021
Standard ProxySQL Cluster rev. 0.4.0906 -- ProxySQL_Cluster.cpp -- Wed Jan 20 14:30:00 2021
Standard ProxySQL Statistics rev. 1.4.1027 -- ProxySQL_Statistics.cpp -- Wed Jan 20 14:30:00 2021
Standard ProxySQL HTTP Server Handler rev. 1.4.1031 -- ProxySQL_HTTP_Server.cpp -- Wed Jan 20 14:30:00 2021
2021-04-15 03:24:40 ProxySQL_Admin.cpp:5539:flush_mysql_variables___database_to_runtime(): [WARNING] Impossible to set variable server_capabilities with value "569867". Resetting to current "569899".
Standard ProxySQL Admin rev. 2.0.6.0805 -- ProxySQL_Admin.cpp -- Wed Jan 20 14:30:00 2021
2021-04-15 03:24:40 [INFO] ProxySQL SHA1 checksum: ab2546523db645833192f38249d9f7df2beb8f78
Standard MySQL Threads Handler rev. 0.2.0902 -- MySQL_Thread.cpp -- Wed Jan 20 14:30:00 2021
Standard MySQL Authentication rev. 0.2.0902 -- MySQL_Authentication.cpp -- Wed Jan 20 14:30:00 2021
2021-04-15 03:24:40 [INFO] Dumping mysql_servers_incoming
+--------------+------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname | port | gtid_port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 1 | 10.6.4.3 | 3306 | 0 | 1 | 0 | 0 | 4000 | 0 | 0 | 0 | |
| 1 | 10.1.11.59 | 3306 | 0 | 1 | 0 | 0 | 4000 | 0 | 0 | 0 | |

I have core dumps as well to share if needed. Please let me know for any further clarification required.

[aakashmuthuramalingam]

The complete core dump output

GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-120.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/...
Reading symbols from /usr/bin/proxysql...done.
[New LWP 133743]
[New LWP 133744]
[New LWP 133746]
[New LWP 133759]
[New LWP 133750]
[New LWP 133748]
[New LWP 133756]
[New LWP 133745]
[New LWP 133761]
[New LWP 133749]
[New LWP 133747]
[New LWP 133757]
[New LWP 134342]
[New LWP 133762]
[New LWP 133717]
[New LWP 133758]
[New LWP 133764]
[New LWP 133740]
[New LWP 133716]
[New LWP 133741]
[New LWP 133712]
[New LWP 133760]
[New LWP 133742]
[New LWP 133763]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/bin/proxysql --idle-threads -c /etc/proxysql.cnf'.
Program terminated with signal 6, Aborted.
#0 0x00007fefbc111387 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install proxysql-2.0.17-1.x86_64
(gdb)
(gdb) bt full
#0 0x00007fefbc111387 in raise () from /lib64/libc.so.6
No symbol table info available.
#1 0x00007fefbc112a78 in abort () from /lib64/libc.so.6
No symbol table info available.
#2 0x00007fefbc10a1a6 in __assert_fail_base () from /lib64/libc.so.6
No symbol table info available.
#3 0x00007fefbc10a252 in __assert_fail () from /lib64/libc.so.6
No symbol table info available.
#4 0x000000000053ee80 in MySQL_Protocol::generate_pkt_OK (this=0x7fefb4df1720, send=send@entry=true, ptr=ptr@entry=0x0, len=len@entry=0x0, sequence_id=,
affected_rows=affected_rows@entry=0, last_insert_id=0, status=2, warnings=0, msg=0x0) at MySQL_Protocol.cpp:711
last_insert_id_prefix = 0 '\000'
msg_prefix = 0 '\000'
msg_len_len = 1 '\001'
gtid_prefix = 0 '\000'
gtid_len_len =
PRETTY_FUNCTION = "bool MySQL_Protocol::generate_pkt_OK(bool, void**, unsigned int*, uint8_t, unsigned int, uint64_t, uint16_t, uint16_t, char*)"
affected_rows_prefix = 65 'A'
affected_rows_len =
myhdr = {pkt_length = , pkt_id = 2}
size =
last_insert_id_len =
msg_len =
client_session_track =
gtid_len =
_ptr = 0x7fefb5670ab0 "\a"
l =
internal_status =
#5 0x000000000052605f in MySQL_Session::MySQL_Result_to_MySQL_wire (this=this@entry=0x7fefb4d91e00, mysql=0x7fefb18e0f00, MyRS=, _myds=myds@entry=0x0)
at MySQL_Session.cpp:6221
num_rows = 0
nTrx =
setStatus =
myerrno =
PRETTY_FUNCTION = "void MySQL_Session::MySQL_Result_to_MySQL_wire(MYSQL*, MySQL_ResultSet*, MySQL_Data_Stream*)"
#6 0x0000000000538000 in MySQL_Session::handler (this=this@entry=0x7fefb4d91e00) at MySQL_Session.cpp:3896
qdt =
---Type to continue, or q to quit---
myconn = 0x7fefb1c12100
myds = 0x7fefb4df2300
rc =
begint = {tv_sec = 140667507835072, tv_nsec = 141741986256}
endt = {tv_sec = 206158430208, tv_nsec = 48}
prepared_stmt_with_no_params = false
pkt = {ptr = 0x7fefb4ab0230, size = 97}
c =
PRETTY_FUNCTION = "int MySQL_Session::handler()"
func = "handler"
wrong_pass = false
#7 0x0000000000516bac in MySQL_Thread::process_all_sessions (this=this@entry=0x7fefb5600000) at MySQL_Thread.cpp:4450
sess = 0x7fefb4d91e00
total_active_transactions = 0
idle_maintenance_thread = false
sess_sort =
func = "process_all_sessions"
n = 4
rc =
#8 0x000000000051db5c in MySQL_Thread::run (this=this@entry=0x7fefb5600000) at MySQL_Thread.cpp:4134
num_idles =
ttw =
maintenance_interval =
idle_maintenance_thread =
PRETTY_FUNCTION = "void MySQL_Thread::run()"
func = "run"
n =
rc =
#9 0x00000000004ceefc in mysql_worker_thread_func (arg=0x7fefbaf82ed0) at main.cpp:760
thread_attr = {__size = '\000' <repeats 17 times>, "\020", '\000' <repeats 37 times>, __align = 0}
tmp_stack_size = 8388608
mysql_thread = 0x7fefbaf82ed0
worker = 0x7fefb5600000
#10 0x00007fefbd2f1ea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#11 0x00007fefbc1d98dd in clone () from /lib64/libc.so.6
---Type to continue, or q to quit---
No symbol table info available.

[JavierJF]

Hi @aakashmuthuramalingam,

we will need the the coredumps in order to continue to troubleshoot this issue. If you could please share them with us, I will followup on this. Thank you.