crashing a lot :( 2.0.5

[starouille]

Hello,

for a while my proxysql server is crashing sometimes (random?) with a query maybe but i don't know which query (we got > 10Kreq/s easily).

OS: Ubuntu 18.04.2
Proxysql version: 2.0.5 up to date . (backend is mariadb 10.2 up to date)

Happend every day:

Extract of logs to see crash periodicity (detailed logs is below)

2019-07-09 15:11:11 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2019-07-09 23:01:08 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2019-07-10 02:01:27 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2019-07-10 02:54:39 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2019-07-10 05:40:20 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2019-07-10 12:26:52 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2019-07-10 18:00:03 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2019-07-11 12:53:14 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2019-07-11 13:30:44 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2019-07-11 22:00:43 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2019-07-12 18:30:27 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2019-07-12 21:58:44 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2019-07-13 21:18:44 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2019-07-14 03:26:25 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2019-07-14 03:58:31 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2019-07-14 10:10:06 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2019-07-14 13:07:54 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2019-07-14 21:01:06 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2019-07-15 00:09:01 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2019-07-15 01:13:22 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2019-07-15 02:53:21 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2019-07-15 04:14:05 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2019-07-15 04:48:52 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2019-07-15 05:36:37 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2019-07-15 05:57:59 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2019-07-15 17:37:27 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2019-07-16 06:03:16 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2019-07-16 08:00:05 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2019-07-16 11:48:08 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2019-07-16 12:56:45 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2019-07-16 15:48:24 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2019-07-16 16:04:01 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2019-07-16 16:07:08 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2019-07-16 16:10:03 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2019-07-16 16:18:04 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2019-07-16 16:21:04 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2019-07-16 16:26:09 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2019-07-16 16:31:35 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2019-07-16 16:41:27 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!

detailed log:

Standard MySQL Monitor (StdMyMon) rev. 2.0.1226 -- MySQL_Monitor.cpp -- Sat Jun 29 14:11:45 2019
2019-07-16 16:15:00 MySQL_Session.cpp:3446:handler(): [WARNING] Error during query on (0,XXXXX,3306): 1180, Got error 35 "Resource deadlock avoided" during COMMIT
2019-07-16 16:18:04 MySQL_Session.cpp:3446:handler(): [WARNING] Error during query on (0,XXXXX,3306): 1317, Query execution was interrupted
Error: signal 11:
/usr/bin/proxysql(_Z13crash_handleri+0x30)[0x55d425925b90]
/lib/x86_64-linux-gnu/libc.so.6(+0x3ef20)[0x7f33e1132f20]
/usr/bin/proxysql(_ZN13MySQL_Session7handlerEv+0x301a)[0x55d42597d95a]
/usr/bin/proxysql(_ZN12MySQL_Thread20process_all_sessionsEv+0x180)[0x55d42595e760]
/usr/bin/proxysql(_ZN12MySQL_Thread3runEv+0xa71)[0x55d425966a31]
/usr/bin/proxysql(_Z24mysql_worker_thread_funcPv+0x7c)[0x55d42592229c]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f33e22376db]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f33e121588f]
2019-07-16 16:18:04 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2019-07-16 16:18:04 [INFO] ProxySQL version 2.0.5-37-gc8e32eed

[...]

2019-07-16 16:35:52 MySQL_Session.cpp:3446:handler(): [WARNING] Error during query on (0,XXXXX,3306): 1062, Duplicate entry '1563285600-esol7lm9sdru-720' for key 'time'
2019-07-16 16:37:47 MySQL_Session.cpp:3446:handler(): [WARNING] Error during query on (0,XXXXX,3306): 1062, Duplicate entry '1563285600-ienmad6z48tu-360' for key 'time'
2019-07-16 16:40:19 MySQL_Session.cpp:3446:handler(): [WARNING] Error during query on (0,XXXXX,3306): 1180, Got error 35 "Resource deadlock avoided" during COMMIT
2019-07-16 16:40:33 MySQL_Session.cpp:3446:handler(): [WARNING] Error during query on (0,XXXXX,3306): 1062, Duplicate entry '1563285600-lu64s3vd5awp-480' for key 'time'
2019-07-16 16:41:26 MySQL_Session.cpp:3446:handler(): [WARNING] Error during query on (0,XXXXX,3306): 1317, Query execution was interrupted
Error: signal 11:
/usr/bin/proxysql(_Z13crash_handleri+0x30)[0x55d425925b90]
/lib/x86_64-linux-gnu/libc.so.6(+0x3ef20)[0x7f33e1132f20]
/usr/bin/proxysql(_ZN13MySQL_Session7handlerEv+0x301a)[0x55d42597d95a]
/usr/bin/proxysql(_ZN12MySQL_Thread20process_all_sessionsEv+0x180)[0x55d42595e760]
/usr/bin/proxysql(_ZN12MySQL_Thread3runEv+0xa71)[0x55d425966a31]
/usr/bin/proxysql(_Z24mysql_worker_thread_funcPv+0x7c)[0x55d42592229c]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f33e22376db]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f33e121588f]
2019-07-16 16:41:27 main.cpp:1373:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!

i can attach a compressed full logs but only in private (sensitive informations).

no core dump generated as i see :(.

Do you have any ideas?
Thank you!

[leagueofsoups]

May be this?
#2133
If you try manipulate auto commit, update to 2.0.6. require. Error like my on 2.0.5.

[starouille]

not only :(, i've this issue from 2.x, sometimes 1-2 per day but yes since 2.0.5 it's 10-20 per day :(

[renecannao]

Try to enable core dumps.
If it is crashing so frequently it means it is easy to reproduce thus also possibly easy to fix.

[starouille]

hello,

i've a core dump.

how can i send you in private? thank you!

EDIT: sent.

[renecannao]

@starouille : thank you for the core dump.

(gdb) bt
#0  MySQL_Session::handler (this=this@entry=0x7f9574e45a80) at MySQL_Session.cpp:3548
#1  0x00005607abb99760 in MySQL_Thread::process_all_sessions (this=this@entry=0x7f9574dff000) at MySQL_Thread.cpp:3726
#2  0x00005607abba1a31 in MySQL_Thread::run (this=0x7f9574dff000) at MySQL_Thread.cpp:3475
#3  0x00005607abb5d29c in mysql_worker_thread_func (arg=0x7f957b027ea0) at main.cpp:627
#4  0x00007f957c60e6db in start_thread (arg=0x7f957657a700) at pthread_create.c:463
#5  0x00007f957b5ec88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb) l
3543                                                                            }
3544                                                                            break;
3545                                                                    case PROCESSING_STMT_EXECUTE:
3546                                                                            {
3547                                                                                    char sqlstate[10];
3548                                                                                    if (myconn->mysql) {
3549                                                                                            sprintf(sqlstate,"%s",mysql_sqlstate(myconn->mysql));
3550                                                                                            client_myds->myprot.generate_pkt_ERR(true,NULL,NULL,client_myds->pkt_sid+1,mysql_errno(myconn->mysql),sqlstate,(char *)mysql_stmt_error(myconn->query.stmt));
3551                                                                                    } else {
3552                                                                                            client_myds->myprot.generate_pkt_ERR(true,NULL,NULL,client_myds->pkt_sid+1, 2013, (char *)"HY000" ,(char *)"Lost connection to MySQL server during query");
(gdb) p myerr 
$12 = 1317

This core dump points to issue #2131 (code line in 2.0.4 and 2.0.5 differ tho), already fixed in 2.0.6 .

Can you please open more core dumps with gdb, and check if all the backtrace (bt in gdb) match the above?
If some core dump has a different backtrace, please send it over.
Thanks

[starouille]

hello

thank you,

i've cheked last 4 coredump and it's always the same backtrace.

Thank you for your work again, i will close this issue as #2131 already exists.