Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ProxySQL 2.0.10 crashing due to signal 11 #3690

Closed
kishmu419 opened this issue Nov 10, 2021 · 8 comments · Fixed by #3710
Closed

ProxySQL 2.0.10 crashing due to signal 11 #3690

kishmu419 opened this issue Nov 10, 2021 · 8 comments · Fixed by #3710

Comments

@kishmu419
Copy link

ProxySQL 2.0.10 crashed with Signal 11.

ProxySQL version - 2.0.10-27-g5b31997
OS version - Ubuntu 16.04.6 LTS

Proxysql error log :

2021-11-09 13:00:01 MySQL_Thread.cpp:4583:process_all_sessions(): [WARNING] Closing unhealthy client connection 172.33.96.38:27526
Error: signal 11:
proxysql(_Z13crash_handleri+0x2d)[0x4ae7fd]
/lib/x86_64-linux-gnu/libc.so.6(+0x354c0)[0x7f5df8df24c0]
proxysql(_ZN12MySQL_Thread20process_all_sessionsEv+0x599)[0x4f3299]
proxysql(_ZN12MySQL_Thread3runEv+0x16e8)[0x4fb258]
proxysql(_Z24mysql_worker_thread_funcPv+0x7c)[0x4a740c]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f5df9e3b6ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f5df8ec44dd]
2021-11-09 13:00:06 main.cpp:1538:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2021-11-09 13:00:06 [INFO] ProxySQL version 2.0.10-27-g5b31997
2021-11-09 13:00:06 [INFO] ProxySQL SHA1 checksum: a9d97b741a0eb29deca65410a4e82319d1b4f2ce
2021-11-09 13:00:06 [INFO] Angel process started ProxySQL process 6805
Standard ProxySQL MySQL Logger rev. 2.0.0714 -- MySQL_Logger.cpp -- Mon Feb 24 16:38:57 2020
Standard ProxySQL Cluster rev. 0.4.0906 -- ProxySQL_Cluster.cpp -- Mon Feb 24 16:38:57 2020
Standard ProxySQL Statistics rev. 1.4.1027 -- ProxySQL_Statistics.cpp -- Mon Feb 24 16:38:57 2020
Standard ProxySQL HTTP Server Handler rev. 1.4.1031 -- ProxySQL_HTTP_Server.cpp -- Mon Feb 24 16:38:57 2020
2021-11-09 13:00:06 [INFO] Switching query logging format from 1 to 2
Standard ProxySQL Admin rev. 2.0.6.0805 -- ProxySQL_Admin.cpp -- Mon Feb 24 16:38:57 2020
2021-11-09 13:00:06 [INFO] ProxySQL SHA1 checksum: a9d97b741a0eb29deca65410a4e82319d1b4f2ce
Standard MySQL Threads Handler rev. 0.2.0902 -- MySQL_Thread.cpp -- Mon Feb 24 16:38:57 2020
Standard MySQL Authentication rev. 0.2.0902 -- MySQL_Authentication.cpp -- Mon Feb 24 16:38:57 2020
2021-11-09 13:00:06 [INFO] Dumping mysql_servers_incoming

Core dump :

GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/proxysql...done.
[New LWP 1705]
[New LWP 1694]
[New LWP 1696]
[New LWP 1697]
[New LWP 1700]
[New LWP 1698]
[New LWP 1701]
[New LWP 1699]
[New LWP 1702]
[New LWP 1703]
[New LWP 1711]
[New LWP 1706]
[New LWP 1712]
[New LWP 1713]
[New LWP 1735]
[New LWP 1736]
[New LWP 1704]
[New LWP 1707]
[New LWP 1740]
[New LWP 1709]
[New LWP 1822]
[New LWP 1826]
[New LWP 1858]
[New LWP 1708]
[New LWP 6800]
[New LWP 1741]
[New LWP 1743]
[New LWP 1739]
[New LWP 3490]
[New LWP 1738]
[New LWP 1742]
[New LWP 1859]
[New LWP 1737]
[New LWP 1710]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `proxysql -c /etc/proxysql.cnf -D /var/lib/proxysql'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 MySQL_Thread::process_all_sessions (this=this@entry=0x7f5df0000000) at MySQL_Thread.cpp:4487
4487 MySQL_Thread.cpp: No such file or directory.
[Current thread is 1 (Thread 0x7f5df61eb700 (LWP 1705))]
(gdb) bt full
#0 MySQL_Thread::process_all_sessions (this=this@entry=0x7f5df0000000) at MySQL_Thread.cpp:4487
a = 0
n = 37
total_active_transactions_ = 0
idle_maintenance_thread = false
rc =
func = "process_all_sessions"
#1 0x00000000004fb258 in MySQL_Thread::run (this=this@entry=0x7f5df0000000) at MySQL_Thread.cpp:4327
num_idles =
ttw =
maintenance_interval =
n =
rc =
idle_maintenance_thread =
PRETTY_FUNCTION = "void MySQL_Thread::run()"
func = "run"
#2 0x00000000004a740c in mysql_worker_thread_func (arg=0x7f5df78a2f10) at main.cpp:743
thread_attr = {__size = '\000' <repeats 17 times>, "\020", '\000' <repeats 37 times>, __align = 0}
tmp_stack_size = 8388608
mysql_thread = 0x7f5df78a2f10
worker = 0x7f5df0000000
#3 0x00007f5df9e3b6ba in start_thread (arg=0x7f5df61eb700) at pthread_create.c:333
_res =
pd = 0x7f5df61eb700
now =
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140041537894144, 308842705401010586, 0, 140725267269407, 140041537894848, 140041515671552, -363992290258724454, -364007572692280934}, mask_was_saved = 0}},
priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call =
pagesize_m1 =
sp =
freesize =
PRETTY_FUNCTION = "start_thread"
#4 0x00007f5df8ec44dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
No locals.
(gdb)
#0 MySQL_Thread::process_all_sessions (this=this@entry=0x7f5df0000000) at MySQL_Thread.cpp:4487
a = 0
n = 37
total_active_transactions
= 0
idle_maintenance_thread = false
rc =
func = "process_all_sessions"
#1 0x00000000004fb258 in MySQL_Thread::run (this=this@entry=0x7f5df0000000) at MySQL_Thread.cpp:4327
num_idles =
ttw =
maintenance_interval =
n =
rc =
idle_maintenance_thread =
PRETTY_FUNCTION = "void MySQL_Thread::run()"
func = "run"
#2 0x00000000004a740c in mysql_worker_thread_func (arg=0x7f5df78a2f10) at main.cpp:743
thread_attr = {__size = '\000' <repeats 17 times>, "\020", '\000' <repeats 37 times>, __align = 0}
tmp_stack_size = 8388608
mysql_thread = 0x7f5df78a2f10
worker = 0x7f5df0000000
#3 0x00007f5df9e3b6ba in start_thread (arg=0x7f5df61eb700) at pthread_create.c:333
__res =
pd = 0x7f5df61eb700
now =
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140041537894144, 308842705401010586, 0, 140725267269407, 140041537894848, 140041515671552, -363992290258724454, -364007572692280934}, mask_was_saved = 0}},
priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call =
pagesize_m1 =
sp =
freesize =
PRETTY_FUNCTION = "start_thread"
#4 0x00007f5df8ec44dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
No locals.

@renecannao
Copy link
Contributor

Please upgrade

@kishmu419
Copy link
Author

kishmu419 commented Nov 18, 2021

Dear Renecanno,

Still proxysql is crashing even after upgraded to the latest version 2.3.2-10-g8cd66cf.

OS version - Ubuntu 16.04.6 LTS
Proxysql version : 2.3.2-10-g8cd66cf.

Proxysql error log :

Error: signal 11:
proxysql(_Z13crash_handleri+0x3d)[0x5f7cbd]
/lib/x86_64-linux-gnu/libc.so.6(+0x354c0)[0x7ff34098c4c0]
/lib/x86_64-linux-gnu/libc.so.6(+0x14e295)[0x7ff340aa5295]
proxysql(_ZN14MySQL_Protocol16generate_pkt_rowEbPPvPjhiPmPPc+0x143)[0x6828c3]
proxysql(_ZN13MySQL_Session16SQLite3_to_MySQLEP14SQLite3_resultPciP14MySQL_Protocolbb+0x1cc)[0x664ccc]
proxysql(_Z21admin_session_handlerP13MySQL_SessionPvP10_PtrSize_t+0x1c4a)[0x6d420a]
proxysql(_ZN13MySQL_Session80handler___status_WAITING_CLIENT_DATA___STATE_SLEEP___MYSQL_COM_QUERY___not_mysqlER10_PtrSize_t+0x2f)[0x661adf]
proxysql(_ZN13MySQL_Session20get_pkts_from_clientERbR10_PtrSize_t+0x8f9)[0x67c799]
proxysql(_ZN13MySQL_Session7handlerEv+0xc7)[0x67a787]
proxysql(_Z11child_mysqlPv+0x3a2)[0x696e22]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7ff3419d56ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7ff340a5e4dd]
 ---- proxysql(_Z13crash_handleri+0x3d) [0x5f7cbd] : crash_handler(int)
 ---- proxysql(_ZN14MySQL_Protocol16generate_pkt_rowEbPPvPjhiPmPPc+0x143) [0x6828c3] : MySQL_Protocol::generate_pkt_row(bool, void**, unsigned int*, unsigned char, int, unsigned long*, char**)
 ---- proxysql(_ZN13MySQL_Session16SQLite3_to_MySQLEP14SQLite3_resultPciP14MySQL_Protocolbb+0x1cc) [0x664ccc] : MySQL_Session::SQLite3_to_MySQL(SQLite3_result*, char*, int, MySQL_Protocol*, bool, bool)
 ---- proxysql(_Z21admin_session_handlerP13MySQL_SessionPvP10_PtrSize_t+0x1c4a) [0x6d420a] : admin_session_handler(MySQL_Session*, void*, _PtrSize_t*)
 ---- proxysql(_ZN13MySQL_Session20get_pkts_from_clientERbR10_PtrSize_t+0x8f9) [0x67c799] : MySQL_Session::get_pkts_from_client(bool&, _PtrSize_t&)
 ---- proxysql(_ZN13MySQL_Session7handlerEv+0xc7) [0x67a787] : MySQL_Session::handler()
 ---- proxysql(_Z11child_mysqlPv+0x3a2) [0x696e22] : child_mysql(void*)
2021-11-18 09:45:04 main.cpp:1245:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2021-11-18 09:45:04 [INFO] ProxySQL version 2.3.2-10-g8cd66cf
2021-11-18 09:45:04 [INFO] ProxySQL SHA1 checksum: 3e37fdfe543b79b7b54099f2d88e741e38ac1597
2021-11-18 09:45:04 [INFO] Angel process started ProxySQL process 13325

Coredump :

GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/proxysql...done.
[New LWP 13324]
[New LWP 9353]
[New LWP 9355]
[New LWP 9356]
[New LWP 9357]
[New LWP 9358]
[New LWP 9360]
[New LWP 9362]
[New LWP 9361]
[New LWP 9368]
[New LWP 9359]
[New LWP 9365]
[New LWP 9372]
[New LWP 9363]
[New LWP 9367]
[New LWP 9369]
[New LWP 9371]
[New LWP 9370]
[New LWP 9373]
[New LWP 9374]
[New LWP 9380]
[New LWP 9383]
[New LWP 9364]
[New LWP 9377]
[New LWP 9384]
[New LWP 9385]
[New LWP 9389]
[New LWP 9366]
[New LWP 9395]
[New LWP 9382]
[New LWP 9386]
[New LWP 9381]
[New LWP 9387]
[New LWP 9397]
[New LWP 9376]
[New LWP 9396]
[New LWP 9379]
[New LWP 10052]
[New LWP 9388]
[New LWP 10048]
[New LWP 9378]
[New LWP 8145]
[New LWP 10051]
[New LWP 10049]
[New LWP 27620]
[New LWP 10050]
[New LWP 10053]
[New LWP 9375]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `proxysql --idle-threads -c /etc/proxysql.cnf -D /var/lib/proxysql'.
****Program terminated with signal SIGSEGV, Segmentation fault.****
#0  0x00007ff340aa5295 in __memcpy_avx_unaligned () at ../sysdeps/x86_64/multiarch/memcpy-avx-unaligned.S:254
254	../sysdeps/x86_64/multiarch/memcpy-avx-unaligned.S: No such file or directory.
[Current thread is 1 (Thread 0x7ff3289ff700 (LWP 13324))]
(gdb) bt full
#0  0x00007ff340aa5295 in __memcpy_avx_unaligned () at ../sysdeps/x86_64/multiarch/memcpy-avx-unaligned.S:254
No locals.
#1  0x00000000006828c3 in memcpy (__len=44461305, __src=<optimized out>, __dest=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/string3.h:53
No locals.
#2  write_encoded_length_and_string (string=<optimized out>, prefix=<optimized out>, len=<optimized out>, val=44461305, p=<optimized out>) at MySQL_Protocol.cpp:132
No locals.
#3  MySQL_Protocol::generate_pkt_row (this=this@entry=0x7ff327c1d0a0, send=send@entry=true, ptr=ptr@entry=0x0, len=len@entry=0x0, sequence_id=<optimized out>, colnums=<optimized out>, fieldslen=0x7ff327dfdd00, 
    fieldstxt=0x7ff327c0cb80) at MySQL_Protocol.cpp:975
        col = 13
        rowlen = <optimized out>
        size = <optimized out>
        _ptr = <optimized out>
        l = 107
#4  0x0000000000664ccc in MySQL_Session::SQLite3_to_MySQL (this=this@entry=0x7ff327c0f000, result=0x7ff327c6e000, error=error@entry=0x0, affected_rows=<optimized out>, myprot=0x7ff327c1d0a0, 
    in_transaction=in_transaction@entry=false, deprecate_eof_active=false) at MySQL_Session.cpp:6606
        r = <optimized out>
        nTrx = <optimized out>
        setStatus = 2
        p = 0x7ff327c0cb80
        l = 0x7ff327dfdd00
        __PRETTY_FUNCTION__ = "void MySQL_Session::SQLite3_to_MySQL(SQLite3_result*, char*, int, MySQL_Protocol*, bool, bool)"
        sid = <optimized out>
#5  0x00000000006d420a in admin_session_handler (sess=<optimized out>, _pa=0x7ff33f25cf00, pkt=<optimized out>) at ProxySQL_Admin.cpp:4782
        SPA = 0x7ff33f25cf00
        pa = 0x7ff33f25cf00
        needs_vacuum = true
        error = 0x0
        cols = 16
        affected_rows = 0
        run_query = <optimized out>
        resultset = 0x7ff327c6e000
        strA = <optimized out>
        strB = <optimized out>
        strAl = <optimized out>
        strBl = <optimized out>
        query = 0x7ff327c150f0 "select * from stats_mysql_processlist"
        query_length = 38
        query_no_space = <optimized out>
        query_no_space_length = <optimized out>
        __func__ = "admin_session_handler"
#6  0x0000000000661adf in MySQL_Session::handler___status_WAITING_CLIENT_DATA___STATE_SLEEP___MYSQL_COM_QUERY___not_mysql (this=this@entry=0x7ff327c0f000, pkt=...) at MySQL_Session.cpp:3201
        __PRETTY_FUNCTION__ = "void MySQL_Session::handler___status_WAITING_CLIENT_DATA___STATE_SLEEP___MYSQL_COM_QUERY___not_mysql(PtrSize_t&)"
#7  0x000000000067c799 in MySQL_Session::get_pkts_from_client (this=this@entry=0x7ff327c0f000, wrong_pass=@0x7ff3289fcfa3: false, pkt=...) at MySQL_Session.cpp:3670
        handler_ret = 0
        c = <optimized out>
        __func__ = "get_pkts_from_client"
        __PRETTY_FUNCTION__ = "int MySQL_Session::get_pkts_from_client(bool&, PtrSize_t&)"
#8  0x000000000067a787 in MySQL_Session::handler (this=this@entry=0x7ff327c0f000) at MySQL_Session.cpp:4254
        handler_ret = 0
---Type <return> to continue, or q <return> to quit---return
        prepared_stmt_with_no_params = false
        wrong_pass = false
        pkt = {ptr = 0x7ff327c150c0, size = 42}
        __PRETTY_FUNCTION__ = "int MySQL_Session::handler()"
        __func__ = "handler"
#9  0x0000000000696e22 in child_mysql (arg=<optimized out>) at ProxySQL_Admin.cpp:4935
        rc = <optimized out>
        thread_attr = {__size = '\000' <repeats 17 times>, "\020", '\000' <repeats 37 times>, __align = 0}
        tmp_stack_size = 8388608
        myarg = <optimized out>
        client = <optimized out>
        fds = {{fd = 892, events = 1, revents = 1}}
        nfds = 1
        rc = <optimized out>
        mysql_thr = 0x7ff327c64000
        sess = 0x7ff327c0f000
        myds = 0x7ff327c1d000
#10 0x00007ff3419d56ba in start_thread (arg=0x7ff3289ff700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7ff3289ff700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140682335352576, 1151186366440590190, 0, 140682676653903, 8388608, 140682390033968, -1144049712614683794, -1144100277351298194}, mask_was_saved = 0}}, 
          priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#11 0x00007ff340a5e4dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:74
No locals.
#12 0x0000000000000000 in ?? ()

@renecannao
Copy link
Contributor

Reopening.

Do you have a reproducible test case?
This seems triggered in a connection to Admin.

@renecannao renecannao reopened this Nov 18, 2021
@renecannao
Copy link
Contributor

I am now seeing it was a simple select * from stats_mysql_processlist .
Can you please share the core dump, privately?

@shan1234567
Copy link

Hi Rene,

As requested , I have sent the coredump privately with you .Please check and share your observation

@kishmu419
Copy link
Author

kishmu419 commented Nov 25, 2021

Hello Renecannao,
Could you please update on this.

@JavierJF
Copy link
Collaborator

Hi @kishmu419 and @shan1234567,

thanks to the provided coredump we have being able to identify the issue and we are working on a fix.

@kishmu419
Copy link
Author

kishmu419 commented Nov 30, 2021 via email

JavierJF added a commit that referenced this issue Dec 1, 2021
…#3690

- Provided a new function to partially initialize 'MySQL_ResultSet'.
- Replaced 'generate_pkt_row' in favor of 'generate_pkt_row3' for
  resulset writing.
renecannao added a commit that referenced this issue Dec 6, 2021
Fixed invalid handling of large packets for Admin - Closes #3690
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants