Proxysql crashed | Version 2.0.12 #2889

aakashmuthuramalingam · 2020-06-15T15:29:21Z

Hi,

Proxysql is getting crashed frequently. It is a production environment and this is the server specification.

OS. : Ubuntu 14.04.5 LTS
CPU: 4
Proxysql Version: 1.4.8

The below is my core dump file output

> (gdb) bt full
> #0  0x00007f933f47ec37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
>         resultvar = 0
>         pid = 11711
>         selftid = 11714
> #1  0x00007f933f482028 in __GI_abort () at abort.c:89
>         save_stage = 2
>         act = {__sigaction_handler = {sa_handler = 0x7ffe38cf5f17, sa_sigaction = 0x7ffe38cf5f17}, sa_mask = {__val = {140270399948100, 7598702, 542, 4294967295, 140270398575955,
>               4, 140270426002000, 47244640256, 0, 6145, 0, 0, 0, 21474836480, 140270399947457, 140270426083328}}, sa_flags = 1063043096, sa_restorer = 0x73f3b6}
>         sigs = {__val = {32, 0 <repeats 15 times>}}
> #2  0x00007f933f477bf6 in __assert_fail_base (fmt=0x7f933f5cc018 "C/o Keld Simonsen, Skt. Jorgens Alle 8, DK-1615 Kobenhavn V", assertion=assertion@entry=0x73f3b6 "0",
>     file=file@entry=0x73f26e "MySQL_Protocol.cpp", line=line@entry=542,
>     function=function@entry=0x73f600 <MySQL_Protocol::generate_pkt_ERR(bool, void**, unsigned int*, unsigned char, unsigned short, char*, char*)::__PRETTY_FUNCTION__> "bool MySQL_Protocol::generate_pkt_ERR(bool, void**, unsigned int*, uint8_t, uint16_t, char*, char*)") at assert.c:92
>         str = 0x7f933c424960 "proxysql: MySQL_Protocol.cpp:542: bool MySQL_Protocol::generate_pkt_ERR(bool, void**, unsigned int*, uint8_t, uint16_t, char*, char*): Assertion `0' failed.\n"
>         total = 4096
> #3  0x00007f933f477ca2 in __GI___assert_fail (assertion=0x73f3b6 "0", file=0x73f26e "MySQL_Protocol.cpp", line=542,
>     function=0x73f600 <MySQL_Protocol::generate_pkt_ERR(bool, void**, unsigned int*, unsigned char, unsigned short, char*, char*)::__PRETTY_FUNCTION__> "bool MySQL_Protocol::generate_pkt_ERR(bool, void**, unsigned int*, uint8_t, uint16_t, char*, char*)") at assert.c:101
> No locals.
> #4  0x000000000053137f in MySQL_Protocol::generate_pkt_ERR (this=0x7f9337db0aa0, send=true, ptr=0x0, len=0x0, sequence_id=18 '\022', error_code=2013,
>     sql_state=0x7f9340ea1460 "HY000", sql_message=0x7f9338d8f497 "Lost connection to MySQL server during query") at MySQL_Protocol.cpp:542
>         __PRETTY_FUNCTION__ = "bool MySQL_Protocol::generate_pkt_ERR(bool, void**, unsigned int*, uint8_t, uint16_t, char*, char*)"
>         __func__ = "generate_pkt_ERR"
>         myhdr = {pkt_length = 53, pkt_id = 18}
>         sql_message_len = 44
>         size = 57
>         _ptr = 0x7f93341c3680 "5"
>         l = 13
> #5  0x0000000000528e0c in MySQL_Session::MySQL_Result_to_MySQL_wire (this=0x7f9337af9880, mysql=0x7f9338d8f400, MyRS=0x0, _myds=0x0) at MySQL_Session.cpp:4188
>         sqlstate = "HY000\000\000\000\000"
>         myerrno = 2013
>         __PRETTY_FUNCTION__ = "void MySQL_Session::MySQL_Result_to_MySQL_wire(MYSQL*, MySQL_ResultSet*, MySQL_Data_Stream*)"
> #6  0x000000000052285f in MySQL_Session::handler (this=0x7f9337af9880) at MySQL_Session.cpp:2975
>         myconn = 0x7f9337d20cc0
>         myds = 0x7f933d619800
>         rc = 2
>         begint = {tv_sec = 140270426003072, tv_nsec = 2}
>         endt = {tv_sec = 140269336920064, tv_nsec = 140270274021888}
>         prepared_stmt_with_no_params = false
>         __func__ = "handler"
>         pkt = {ptr = 0x7f9333a52780, size = 77}
>         c = 3 '\003'
>         __PRETTY_FUNCTION__ = "int MySQL_Session::handler()"
>         wrong_pass = false
>         j = 0
> #7  0x000000000050db8f in MySQL_Thread::process_all_sessions (this=0x7f933d60d000) at MySQL_Thread.cpp:3237
>         sess = 0x7f9337af9880
>         total_active_transactions_ = 0
> ---Type <return> to continue, or q <return> to quit---
>         idle_maintenance_thread = false
>         sess_sort = true
>         total_active_transactions_tmp = 1501
>         n = 0
>         rc = 1519614332
> #8  0x000000000050ce6d in MySQL_Thread::run (this=0x7f933d60d000) at MySQL_Thread.cpp:3024
>         num_idles = 23
>         maintenance_interval = 1000000
>         idle_maintenance_thread = false
>         __func__ = "run"
>         __PRETTY_FUNCTION__ = "void MySQL_Thread::run()"
>         n = 47
>         rc = 1
> #9  0x00000000004c815b in mysql_worker_thread_func (arg=0x7f933f037380) at main.cpp:243
>         thread_attr = {__size = '\000' <repeats 17 times>, "\020", '\000' <repeats 37 times>, __align = 0}
>         tmp_stack_size = 8388608
>         mysql_thread = 0x7f933f037380
>         worker = 0x7f933d60d000
> #10 0x00007f9340a80184 in start_thread (arg=0x7f9340ea2700) at pthread_create.c:312
>         __res = <optimized out>
>         pd = 0x7f9340ea2700
>         now = <optimized out>
>         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140270426007296, 2281701945656097320, 0, 0, 140270426008000, 140270426007296, -2266232000821708248, -2266231433679062488},
>               mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
>         not_first_call = <optimized out>
>         pagesize_m1 = <optimized out>
>         sp = <optimized out>
>         freesize = <optimized out>
>         __PRETTY_FUNCTION__ = "start_thread"
> #11 0x00007f933f545ffd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:74
> No locals.
> #12 0x0000000000000000 in ?? ()

This is from my proxysql log file before and after crash

> 2019-11-26 20:56:31 MySQL_Session.cpp:2824:handler(): [WARNING] Retrying query.
> 2019-11-26 20:56:33 MySQL_Thread.cpp:3080:process_data_on_data_stream(): [WARNING] Detected broken idle connection on :3306
> 2019-11-26 20:56:41 MySQL_Thread.cpp:3080:process_data_on_data_stream(): [WARNING] Detected broken idle connection on :3306
> Error: signal 11:
> proxysql(_Z13crash_handleri+0x25)[0x4cd844]
> /lib/x86_64-linux-gnu/libc.so.6(+0x36cb0)[0x7f21b8b9acb0]
> /lib/x86_64-linux-gnu/libc.so.6(strlen+0x2a)[0x7f21b8becdfa]
> /lib/x86_64-linux-gnu/libc.so.6(__strdup+0xe)[0x7f21b8becb3e]
> proxysql(_ZN21QP_query_digest_stats7get_rowEv+0x78)[0x53f5d0]
> proxysql(_ZN15Query_Processor17get_query_digestsEv+0x1d1)[0x539209]
> proxysql(_ZN14ProxySQL_Admin27stats___mysql_query_digestsEb+0x5e)[0x55c7e4]
> proxysql(_ZN14ProxySQL_Admin24GenericRefreshStatisticsEPKcjb+0x3f2)[0x54cef2]
> proxysql(_Z21admin_session_handlerP13MySQL_SessionPvP10_PtrSize_t+0x3d7)[0x54e176]
> proxysql(_ZN13MySQL_Session7handlerEv+0xf91)[0x51ec8f]
> proxysql(_Z11child_mysqlPv+0x2d2)[0x551ebc]
> /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184)[0x7f21ba19c184]
> /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f21b8c6203d]
> 2019-11-26 20:56:47 main.cpp:905:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
> 2019-11-26 20:56:47 main.cpp:877:ProxySQL_daemonize_phase3(): [INFO] Angel process started ProxySQL process 4654
> Standard ProxySQL Cluster rev. 0.1.0702_DEBUG -- ProxySQL_Cluster.cpp -- Tue Apr  3 06:28:45 2018
> Standard ProxySQL Statistics rev. 1.4.1027_DEBUG -- ProxySQL_Statistics.cpp -- Wed Apr  4 13:21:29 2018
> Standard ProxySQL HTTP Server Handler rev. 1.4.1031_DEBUG -- ProxySQL_HTTP_Server.cpp -- Tue Apr  3 06:28:45 2018
> Admin initialized in 0.015388 secs.
> Standard ProxySQL Admin rev. 0.2.0902_DEBUG -- ProxySQL_Admin.cpp -- Tue Apr 10 12:18:22 2018
> Standard MySQL Threads Handler rev. 0.2.0902_DEBUG -- MySQL_Thread.cpp -- Tue Apr  3 06:28:45 2018
> Main phase3 : GloMyLogger initialized in 3e-06 secs.
> Standard MySQL Authentication rev. 0.2.0902_DEBUG -- MySQL_Authentication.cpp -- Tue Apr  3 06:28:45 2018
> Main init phase2 completed in 0.021967 secs.
> Main phase3 : GloMyLogger initialized in 3e-06 secs.

Kindly do let me know on further details.

The text was updated successfully, but these errors were encountered:

renecannao · 2020-06-15T15:32:28Z

You need to upgrade.
This looks like a known bug.

aakashmuthuramalingam · 2020-06-15T15:35:47Z

Yes I upgraded to the version 2.0.12. It repeats. It is still getting crashed.

Logs:

Error: signal 11:
/usr/bin/proxysql(_Z13crash_handleri+0x34)[0x55b1a993785e]
/lib/x86_64-linux-gnu/libc.so.6(+0x3ef20)[0x7f2a09f21f20]
/usr/bin/proxysql(_ZN14MySQL_Protocol25generate_pkt_column_countEbPPvPjhmP15MySQL_ResultSet+0x17e)[0x55b1a99f3f38]
/usr/bin/proxysql(_ZN15MySQL_ResultSet4initEP14MySQL_ProtocolP12st_mysql_resP8st_mysqlP13st_mysql_stmt+0x212)[0x55b1a99fa8c4]
/usr/bin/proxysql(_ZN16MySQL_Connection7handlerEs+0x164d)[0x55b1a9b9b029]
/usr/bin/proxysql(_ZN16MySQL_Connection11async_queryEsPcmPP13st_mysql_stmtP23stmt_execute_metadata_t+0x254)[0x55b1a9b9c388]
/usr/bin/proxysql(_ZN13MySQL_Session7handlerEv+0x3eb3)[0x55b1a99d3f5b]
/usr/bin/proxysql(_ZN12MySQL_Thread20process_all_sessionsEv+0x970)[0x55b1a99b3976]
/usr/bin/proxysql(_ZN12MySQL_Thread3runEv+0x2e78)[0x55b1a99b27e4]
/usr/bin/proxysql(_Z24mysql_worker_thread_funcPv+0xca)[0x55b1a992ccef]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f2a0b0266db]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f2a0a00488f]
2020-06-15 20:03:58 main.cpp:1564:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2020-06-15 20:03:58 main.cpp:1565:ProxySQL_daemonize_phase3(): [INFO] ProxySQL version 2.0.12-38-g58a909a0_DEBUG
2020-06-15 20:03:58 main.cpp:1567:ProxySQL_daemonize_phase3(): [INFO] ProxySQL SHA1 checksum: fc5925cab37d6eafa96e6c51d94f6baae38e6df5
2020-06-15 20:03:58 main.cpp:1536:ProxySQL_daemonize_phase3(): [INFO] Angel process started ProxySQL process 18663
Standard ProxySQL MySQL Logger rev. 2.0.0714_DEBUG -- MySQL_Logger.cpp -- Sun May 17 20:24:24 2020
Standard ProxySQL Cluster rev. 0.4.0906_DEBUG -- ProxySQL_Cluster.cpp -- Sun May 17 20:24:24 2020
Standard ProxySQL Statistics rev. 1.4.1027_DEBUG -- ProxySQL_Statistics.cpp -- Sun May 17 20:24:24 2020

renecannao · 2020-06-15T15:38:03Z

Do you have a core dump?

aakashmuthuramalingam · 2020-06-15T15:39:45Z

No, I am unable to locate it on this new version. I will try to get it on the next crash. I hope It can get it on /var/lib/proxysql directory after it crashed.

renecannao · 2020-06-15T15:43:13Z

yes, the core dump should be in /var/lib/proxysql (the default datadir).
Having few lines of the error log before the crash can be useful too

aakashmuthuramalingam · 2020-06-15T15:53:25Z

Proxysql was crashed just an hour ago, but I could not find the core dump in my datadir.

root@prod:/var/lib/proxysql# ls
proxysql-ca.pem proxysql.log.2 queries.log.00000003 queries.log.00000009 queries.log.00000015 queries.log.00000021 queries.log.00000027 queries.log.00000033
proxysql-cert.pem proxysql.log.3 queries.log.00000004 queries.log.00000010 queries.log.00000016 queries.log.00000022 queries.log.00000028 queries.log.00000034
proxysql-key.pem proxysql.pid queries.log.00000005 queries.log.00000011 queries.log.00000017 queries.log.00000023 queries.log.00000029 queries.log.00000035
proxysql.db proxysql_stats.db queries.log.00000006 queries.log.00000012 queries.log.00000018 queries.log.00000024 queries.log.00000030
proxysql.log queries.log.00000001 queries.log.00000007 queries.log.00000013 queries.log.00000019 queries.log.00000025 queries.log.00000031
proxysql.log.1 queries.log.00000002 queries.log.00000008 queries.log.00000014 queries.log.00000020 queries.log.00000026 queries.log.00000032

Last few lines before the crash

2020-06-15 20:03:09 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:09 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.6.95:36014
2020-06-15 20:03:10 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:10 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.6.95:60579
2020-06-15 20:03:10 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:10 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.6.95:39282
2020-06-15 20:03:11 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:11 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.6.95:40044
2020-06-15 20:03:11 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:11 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.6.95:36027
2020-06-15 20:03:12 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:12 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.6.95:36032
2020-06-15 20:03:13 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:13 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.6.95:36021
2020-06-15 20:03:15 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:15 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.28.242:49358
2020-06-15 20:03:17 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:17 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.6.95:36031
2020-06-15 20:03:17 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:17 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.28.242:49432
2020-06-15 20:03:18 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:18 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.6.95:36047
2020-06-15 20:03:18 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:18 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.6.95:36051
2020-06-15 20:03:18 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.27.106:3306
2020-06-15 20:03:18 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.6.95:46936
2020-06-15 20:03:18 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:18 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.6.95:36048
2020-06-15 20:03:20 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:20 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.6.95:47846
2020-06-15 20:03:22 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:22 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.6.95:36043
2020-06-15 20:03:23 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:23 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.28.242:49308
2020-06-15 20:03:25 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.29.180:3306
2020-06-15 20:03:25 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.28.242:49452
2020-06-15 20:03:28 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:28 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.6.95:43803
2020-06-15 20:03:30 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:30 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.6.95:36024
2020-06-15 20:03:31 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.29.180:3306
2020-06-15 20:03:31 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.28.242:48584
2020-06-15 20:03:32 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:32 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:32 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.6.95:36028
2020-06-15 20:03:33 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:33 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.6.95:37018
2020-06-15 20:03:33 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:33 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.28.242:49414
2020-06-15 20:03:33 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:33 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.28.242:49396
2020-06-15 20:03:36 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:36 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.6.95:36020
2020-06-15 20:03:37 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:37 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.6.95:35991
2020-06-15 20:03:37 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:37 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.28.242:49030
2020-06-15 20:03:37 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:37 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.28.242:49444
2020-06-15 20:03:37 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:37 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.6.95:36000
2020-06-15 20:03:38 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:38 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.6.95:59498
2020-06-15 20:03:38 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:38 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.28.242:49450
2020-06-15 20:03:39 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:39 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.6.95:36029
2020-06-15 20:03:40 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:40 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.28.242:49488
2020-06-15 20:03:42 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:42 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.6.95:36095
2020-06-15 20:03:42 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:42 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.6.95:58778
2020-06-15 20:03:42 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:42 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.6.95:39796
2020-06-15 20:03:43 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:43 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.6.95:36049
2020-06-15 20:03:46 MySQL_Session.cpp:3871:handler(): [WARNING] Error during query on (2,192.168.18.73,3306,601376): 1062, Duplicate entry '2Pi4Xq-pzcohOrxWOM1i1EXKrhtYTFXCKSNBqnTzXlKPG2V7cVOU!-1423040038' for key 'PRIMARY'
2020-06-15 20:03:47 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:47 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.6.95:36111
2020-06-15 20:03:47 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:47 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.6.95:36864
2020-06-15 20:03:49 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:49 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.28.242:49520
2020-06-15 20:03:49 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:49 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.6.95:36097
2020-06-15 20:03:52 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:52 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.6.95:36091
2020-06-15 20:03:53 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.27.106:3306
2020-06-15 20:03:53 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.6.95:44125
2020-06-15 20:03:53 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:53 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.28.242:48538
2020-06-15 20:03:54 MySQL_Thread.cpp:4077:process_data_on_data_stream(): [WARNING] Detected broken idle connection on 192.168.18.73:3306
2020-06-15 20:03:54 MySQL_Thread.cpp:4281:process_all_sessions(): [WARNING] Closing unhealthy client connection 192.168.6.95:36098
Error: signal 11:
/usr/bin/proxysql(_Z13crash_handleri+0x34)[0x55b1a993785e]
/lib/x86_64-linux-gnu/libc.so.6(+0x3ef20)[0x7f2a09f21f20]
/usr/bin/proxysql(_ZN14MySQL_Protocol25generate_pkt_column_countEbPPvPjhmP15MySQL_ResultSet+0x17e)[0x55b1a99f3f38]
/usr/bin/proxysql(_ZN15MySQL_ResultSet4initEP14MySQL_ProtocolP12st_mysql_resP8st_mysqlP13st_mysql_stmt+0x212)[0x55b1a99fa8c4]
/usr/bin/proxysql(_ZN16MySQL_Connection7handlerEs+0x164d)[0x55b1a9b9b029]
/usr/bin/proxysql(_ZN16MySQL_Connection11async_queryEsPcmPP13st_mysql_stmtP23stmt_execute_metadata_t+0x254)[0x55b1a9b9c388]
/usr/bin/proxysql(_ZN13MySQL_Session7handlerEv+0x3eb3)[0x55b1a99d3f5b]
/usr/bin/proxysql(_ZN12MySQL_Thread20process_all_sessionsEv+0x970)[0x55b1a99b3976]
/usr/bin/proxysql(_ZN12MySQL_Thread3runEv+0x2e78)[0x55b1a99b27e4]
/usr/bin/proxysql(_Z24mysql_worker_thread_funcPv+0xca)[0x55b1a992ccef]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f2a0b0266db]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f2a0a00488f]
2020-06-15 20:03:58 main.cpp:1564:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2020-06-15 20:03:58 main.cpp:1565:ProxySQL_daemonize_phase3(): [INFO] ProxySQL version 2.0.12-38-g58a909a0_DEBUG
2020-06-15 20:03:58 main.cpp:1567:ProxySQL_daemonize_phase3(): [INFO] ProxySQL SHA1 checksum: fc5925cab37d6eafa96e6c51d94f6baae38e6df5
2020-06-15 20:03:58 main.cpp:1536:ProxySQL_daemonize_phase3(): [INFO] Angel process started ProxySQL process 18663
Standard ProxySQL MySQL Logger rev. 2.0.0714_DEBUG -- MySQL_Logger.cpp -- Sun May 17 20:24:24 2020
Standard ProxySQL Cluster rev. 0.4.0906_DEBUG -- ProxySQL_Cluster.cpp -- Sun May 17 20:24:24 2020
Standard ProxySQL Statistics rev. 1.4.1027_DEBUG -- ProxySQL_Statistics.cpp -- Sun May 17 20:24:24 2020
Standard ProxySQL HTTP Server Handler rev. 1.4.1031_DEBUG -- ProxySQL_HTTP_Server.cpp -- Sun May 17 20:24:24 2020
Admin initialized in 0.067006 secs.
Standard ProxySQL Admin rev. 2.0.6.0805_DEBUG -- ProxySQL_Admin.cpp -- Sun May 17 20:24:24 2020
2020-06-15 20:03:58 main.cpp:974:ProxySQL_Main_init_Admin_module(): [INFO] ProxySQL SHA1 checksum: fc5925cab37d6eafa96e6c51d94f6baae38e6df5
Standard MySQL Threads Handler rev. 0.2.0902_DEBUG -- MySQL_Thread.cpp -- Sun May 17 20:24:24 2020
Main phase3 : GloMyLogger initialized in 5e-06 secs.
Standard MySQL Authentication rev. 0.2.0902_DEBUG -- MySQL_Authentication.cpp -- Sun May 17 20:24:24 2020
Main init phase2 completed in 0.077396 secs.
Main phase3 : GloMyLogger initialized in 5e-06 secs.

renecannao · 2020-06-15T16:31:08Z

Do you still have the core dump from 1.4.8 ?
If yes, can you run this on gdb?

frame 4
p myds->DSS
p **myds

Thanks

aakashmuthuramalingam · 2020-06-15T16:37:00Z

from 1.4.8 core dump

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `proxysql -c /etc/proxysql.cnf -D /var/lib/proxysql'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007f933f47ec37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb)
(gdb)
(gdb) frame 4
#4 0x000000000053137f in MySQL_Protocol::generate_pkt_ERR (this=0x7f9337db0aa0, send=true, ptr=0x0, len=0x0, sequence_id=18 '\022', error_code=2013,
sql_state=0x7f9340ea1460 "HY000", sql_message=0x7f9338d8f497 "Lost connection to MySQL server during query") at MySQL_Protocol.cpp:542
542 MySQL_Protocol.cpp: No such file or directory.
(gdb) p myds->DSS
$1 = STATE_ERR
(gdb) p **myds
$2 = {queueIN = {buffer = 0x7f93357d66c0, size = 32768, head = 0, tail = 0, partial = 77, pkt = {ptr = 0x0, size = 0}, hdr = {pkt_length = 73, pkt_id = 0}}, pkts_recv = 323,
queueOUT = {buffer = 0x7f9333ff2700, size = 32768, head = 0, tail = 0, partial = 0, pkt = {ptr = 0x0, size = 931}, hdr = {pkt_length = 4140095, pkt_id = 46}}, pkts_sent = 353,
CompPktIN = {pkt = {ptr = 0x0, size = 0}, partial = 0}, CompPktOUT = {pkt = {ptr = 0x0, size = 0}, partial = 0}, myprot = {myds = 0x7f9337af9a20, userinfo = 0x7f93393bb980,
sess = 0x7f9337af9880, dump_pkt = true, current_PreStmt = 0x0, prot_status = 0}, mysql_real_query = {pkt = {ptr = 0x0, size = 0}, QueryPtr = 0x0, QuerySize = 0}, bytes_info = {
bytes_recv = 54103, bytes_sent = 347320}, multi_pkt = {ptr = 0x0, size = 0}, pause_until = 0, wait_until = 0, killed_at = 7308613718830771813, max_connect_time = 0,
PSarrayIN = 0x7f9333ba6f70, PSarrayOUT = 0x7f9333ba6df0, resultset = 0x7f9332228950, resultset_length = 0, mypolls = 0x7f933d60d878, myconn = 0x7f9333720100,
sess = 0x7f9337af9880, mybe = 0x0, ssl = 0x0, client_addr = 0x7f93354cd320, addr = {addr = 0x7f933545a3f0 "172.31.6.95", port = 52480}, proxy_addr = {
addr = 0x7f93322a12d8 "0.0.0.0", port = 6033}, connect_tries = 0, query_retries_on_failure = 0, connect_retries_on_failure = 0, DSS = STATE_ERR, myds_type = MYDS_FRONTEND,
client_addrlen = 16, fd = 40, poll_fds_idx = 35, active_transaction = 758263853, active = 1, status = 825242151, revents = 0, encrypted = false, net_failure = false,
pkt_sid = 17 '\021'}
(gdb)

renecannao · 2020-06-15T16:43:02Z

Thank you.
The error you hit in 1.4.8 is indeed fixed in 2.0.7:
https://github.com/sysown/proxysql/blob/v1.4.8/lib/MySQL_Protocol.cpp#L529-L543
vs
https://github.com/sysown/proxysql/blob/v1.4.8/lib/MySQL_Protocol.cpp#L529-L543
(look for STATE_ERR).
The backtrace of 2.0.12 is perhaps different, so perphas a different issue

aakashmuthuramalingam · 2020-06-15T18:36:22Z

Please do let me know if you have any further logs, also is it a bug in 2.0.12 ?

renecannao · 2020-06-15T18:49:45Z

I realized my previous comment has a copy/paste issue. Rewriting:

The error you hit in 1.4.8 is indeed fixed in 2.0.7:
https://github.com/sysown/proxysql/blob/v1.4.8/lib/MySQL_Protocol.cpp#L529-L543
vs
https://github.com/sysown/proxysql/blob/v2.0.7/lib/MySQL_Protocol.cpp#L568-L583
(look for STATE_ERR).
The backtrace of 2.0.12 is perhaps different, so perhaps a different issue.

also is it a bug in 2.0.12 ?

2.0.12 doesn't have the bug you reported in 1.4.8:
https://github.com/sysown/proxysql/blob/v2.0.12/lib/MySQL_Protocol.cpp#L577-L592

Please do let me know if you have any further logs

I would wait a core dump of 2.0.12 , and then I will be happy to troubleshoot it together.
Thanks

aakashmuthuramalingam · 2020-06-16T07:57:15Z

Proxysql just now crashed 5 mins ago, I do not have any idea about the core file in this version, It is not in datadir. Is there any change in newer versions ? where can I get the core dump ?

Logs:

root@ip-192-168-28-242:/var/lib/proxysql# ls
proxysql-ca.pem proxysql.log.3 queries.log.00000005 queries.log.00000012 queries.log.00000019 queries.log.00000026 queries.log.00000033 queries.log.00000040
proxysql-cert.pem proxysql.pid queries.log.00000006 queries.log.00000013 queries.log.00000020 queries.log.00000027 queries.log.00000034 queries.log.00000041
proxysql-key.pem proxysql_stats.db queries.log.00000007 queries.log.00000014 queries.log.00000021 queries.log.00000028 queries.log.00000035 queries.log.00000042
proxysql.db queries.log.00000001 queries.log.00000008 queries.log.00000015 queries.log.00000022 queries.log.00000029 queries.log.00000036 queries.log.00000043
proxysql.log queries.log.00000002 queries.log.00000009 queries.log.00000016 queries.log.00000023 queries.log.00000030 queries.log.00000037 queries.log.00000044
proxysql.log.1 queries.log.00000003 queries.log.00000010 queries.log.00000017 queries.log.00000024 queries.log.00000031 queries.log.00000038 queries.log.00000045
proxysql.log.2 queries.log.00000004 queries.log.00000011 queries.log.00000018 queries.log.00000025 queries.log.00000032 queries.log.00000039 queries.log.00000046
root@ip-192-168-28-242:/var/lib/proxysql# pwd
/var/lib/proxysql

crash logs:

Error: signal 11:
/usr/bin/proxysql(_Z13crash_handleri+0x34)[0x55b1a993785e]
/lib/x86_64-linux-gnu/libc.so.6(+0x3ef20)[0x7f2a09f21f20]
/usr/bin/proxysql(pvio_socket_async_write+0x5d)[0x55b1a9eb3670]
/usr/bin/proxysql(+0x7d7d38)[0x55b1a9e8cd38]
/usr/bin/proxysql(ma_pvio_write+0xd3)[0x55b1a9e8cedb]
/usr/bin/proxysql(ma_net_real_write+0x1d0)[0x55b1a9eb6bb8]
/usr/bin/proxysql(ma_net_flush+0x73)[0x55b1a9eb64ad]
/usr/bin/proxysql(ma_net_write_command+0x1b5)[0x55b1a9eb67e1]
/usr/bin/proxysql(mthd_my_send_cmd+0x207)[0x55b1a9e812d0]
/usr/bin/proxysql(ma_simple_command+0xd0)[0x55b1a9e814c6]
/usr/bin/proxysql(mysql_real_query+0x9e)[0x55b1a9e8678e]
/usr/bin/proxysql(+0x7f4bd3)[0x55b1a9ea9bd3]
/usr/bin/proxysql(my_context_spawn+0x61)[0x55b1a9eb0893]
2020-06-16 13:15:01 main.cpp:1564:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2020-06-16 13:15:01 main.cpp:1565:ProxySQL_daemonize_phase3(): [INFO] ProxySQL version 2.0.12-38-g58a909a0_DEBUG
2020-06-16 13:15:01 main.cpp:1567:ProxySQL_daemonize_phase3(): [INFO] ProxySQL SHA1 checksum: fc5925cab37d6eafa96e6c51d94f6baae38e6df5
2020-06-16 13:15:01 main.cpp:1536:ProxySQL_daemonize_phase3(): [INFO] Angel process started ProxySQL process 2527
Standard ProxySQL MySQL Logger rev. 2.0.0714_DEBUG -- MySQL_Logger.cpp -- Sun May 17 20:24:24 2020
Standard ProxySQL Cluster rev. 0.4.0906_DEBUG -- ProxySQL_Cluster.cpp -- Sun May 17 20:24:24 2020
Standard ProxySQL Statistics rev. 1.4.1027_DEBUG -- ProxySQL_Statistics.cpp -- Sun May 17 20:24:24 2020
Standard ProxySQL HTTP Server Handler rev. 1.4.1031_DEBUG -- ProxySQL_HTTP_Server.cpp -- Sun May 17 20:24:24 2020
Admin initialized in 0.065895 secs.
Standard ProxySQL Admin rev. 2.0.6.0805_DEBUG -- ProxySQL_Admin.cpp -- Sun May 17 20:24:24 2020
2020-06-16 13:15:01 main.cpp:974:ProxySQL_Main_init_Admin_module(): [INFO] ProxySQL SHA1 checksum: fc5925cab37d6eafa96e6c51d94f6baae38e6df5
Standard MySQL Threads Handler rev. 0.2.0902_DEBUG -- MySQL_Thread.cpp -- Sun May 17 20:24:24 2020
Main phase3 : GloMyLogger initialized in 5e-06 secs.
Standard MySQL Authentication rev. 0.2.0902_DEBUG -- MySQL_Authentication.cpp -- Sun May 17 20:24:24 2020
Main init phase2 completed in 0.075201 secs.
Main phase3 : GloMyLogger initialized in 6e-06 secs.
2020-06-16 13:15:01 MySQL_HostGroups_Manager.cpp:1249:commit(): [INFO] Dumping mysql_servers_incoming

aakashmuthuramalingam changed the title ~~Proxysql crashed | Version 1.4.8~~ Proxysql crashed | Version 2.0.12 Jun 16, 2020

renecannao closed this as completed Jan 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Proxysql crashed | Version 2.0.12 #2889

Proxysql crashed | Version 2.0.12 #2889

aakashmuthuramalingam commented Jun 15, 2020 •

edited by renecannao

Loading

renecannao commented Jun 15, 2020

aakashmuthuramalingam commented Jun 15, 2020

renecannao commented Jun 15, 2020

aakashmuthuramalingam commented Jun 15, 2020

renecannao commented Jun 15, 2020 •

edited

Loading

aakashmuthuramalingam commented Jun 15, 2020

renecannao commented Jun 15, 2020

aakashmuthuramalingam commented Jun 15, 2020

renecannao commented Jun 15, 2020

aakashmuthuramalingam commented Jun 15, 2020

renecannao commented Jun 15, 2020

aakashmuthuramalingam commented Jun 16, 2020 •

edited

Loading

Proxysql crashed | Version 2.0.12 #2889

Proxysql crashed | Version 2.0.12 #2889

Comments

aakashmuthuramalingam commented Jun 15, 2020 • edited by renecannao Loading

renecannao commented Jun 15, 2020

aakashmuthuramalingam commented Jun 15, 2020

renecannao commented Jun 15, 2020

aakashmuthuramalingam commented Jun 15, 2020

renecannao commented Jun 15, 2020 • edited Loading

aakashmuthuramalingam commented Jun 15, 2020

renecannao commented Jun 15, 2020

aakashmuthuramalingam commented Jun 15, 2020

renecannao commented Jun 15, 2020

aakashmuthuramalingam commented Jun 15, 2020

renecannao commented Jun 15, 2020

aakashmuthuramalingam commented Jun 16, 2020 • edited Loading

aakashmuthuramalingam commented Jun 15, 2020 •

edited by renecannao

Loading

renecannao commented Jun 15, 2020 •

edited

Loading

aakashmuthuramalingam commented Jun 16, 2020 •

edited

Loading