Crash when unable to reach a backend, and fast_forward is enabled

[renecannao]

Backtrace:

(gdb) bt full
#0  0x00007fb582cbd625 in raise () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007fb582cbee05 in abort () from /lib64/libc.so.6
No symbol table info available.
#2  0x00007fb582cb674e in __assert_fail_base () from /lib64/libc.so.6
No symbol table info available.
#3  0x00007fb582cb6810 in __assert_fail () from /lib64/libc.so.6
No symbol table info available.
#4  0x0000000000460058 in MySQL_Protocol::generate_pkt_ERR (this=0x7fb4d11fe8a0, send=Unhandled dwarf expression opcode 0xf3
) at MySQL_Protocol.cpp:571
        __PRETTY_FUNCTION__ = "bool MySQL_Protocol::generate_pkt_ERR(bool, void**, unsigned int*, uint8_t, uint16_t, char*, char*)"
        myhdr = <error reading variable myhdr (Unhandled dwarf expression opcode 0xf3)>
        sql_message_len = <value optimized out>
        size = <value optimized out>
        _ptr = 0x7fb4a83f6700 "8"
        l = 13
#5  0x000000000045e05c in MySQL_Session::handler (this=0x7fb4d5e6aa00) at MySQL_Session.cpp:1701
        buf = "Max connect failure while reaching hostgroup 11"
        myerr = 0
        myconn = <value optimized out>
        myds = 0x7fb4cdcd4a00
        rc = <value optimized out>
        st = <value optimized out>
        pkt = {ptr = 0x7fb557eda880, size = 776}
        c = <value optimized out>
        __PRETTY_FUNCTION__ = "int MySQL_Session::handler()"
        __func__ = "handler"
        wrong_pass = false
#6  0x0000000000451f72 in MySQL_Thread::process_all_sessions (this=0x7fb57d40d000) at MySQL_Thread.cpp:2025
        sess = 0x7fb4d5e6aa00
        total_active_transactions_ = 0
        total_active_transactions_tmp = <value optimized out>
        n = 636
        rc = <value optimized out>
#7  0x000000000045429b in MySQL_Thread::run (this=0x7fb57d40d000) at MySQL_Thread.cpp:1905
        num_idles = <value optimized out>
        n = <value optimized out>
        rc = 67
        __PRETTY_FUNCTION__ = "void MySQL_Thread::run()"
        __func__ = "run"
#8  0x0000000000436e94 in mysql_worker_thread_func (arg=0x7fb5816400c0) at main.cpp:163
        mysql_thread = 0x7fb5816400c0
        worker = 0x7fb57d40d000
#9  0x00007fb584438aa1 in start_thread () from /lib64/libpthread.so.0

Info about the session:

(gdb) p session_fast_forward
$18 = true
(gdb) p status
$19 = CONNECTING_SERVER
(gdb) p started_sending_data_to_client
$20 = false
(gdb) p client_myds->status
$21 = 0
(gdb) p client_myds->DSS
$22 = STATE_SLEEP

[grypyrg]

fast_forward users crash in my test environment too, probably for the same reason as this bug (let me know if this is a different bug, then I create a separate issue)

• create a user that is fast_forward
• make sure the user does not authenticate to the database backend
• login on proxysql using that user
• crash of proxysql happens

Log output:

2016-11-09 11:33:03 mysql_connection.cpp:559:handler(): [ERROR] Failed to mysql_real_connect() on mysql1:3306 , FD (Conn:39 , MyDS:39) , 1045: Access denied for user 'test'@'192.168.63.24' (using password: YES).
proxysql: MySQL_Protocol.cpp:572: bool MySQL_Protocol::generate_pkt_ERR(bool, void**, unsigned int*, uint8_t, uint16_t, char*, char*): Assertion `0' failed.
Error: signal 6:
proxysql(_Z13crash_handleri+0x1a)[0x43a8da]
/lib64/libc.so.6(+0x32660)[0x7fb56170d660]
/lib64/libc.so.6(gsignal+0x35)[0x7fb56170d5e5]
/lib64/libc.so.6(abort+0x175)[0x7fb56170edc5]
/lib64/libc.so.6(+0x2b70e)[0x7fb56170670e]
/lib64/libc.so.6(__assert_perror_fail+0x0)[0x7fb5617067d0]
proxysql(_ZN14MySQL_Protocol16generate_pkt_ERREbPPvPjhtPcS3_+0x188)[0x469858]
proxysql(_ZN13MySQL_Session40handler_again___status_CONNECTING_SERVEREPi+0x585)[0x464235]
proxysql(_ZN13MySQL_Session7handlerEv+0x1216)[0x466a06]
proxysql(_ZN12MySQL_Thread20process_all_sessionsEv+0x1a4)[0x454f64]
proxysql(_ZN12MySQL_Thread3runEv+0x1651)[0x45d211]
proxysql(_Z24mysql_worker_thread_funcPv+0x54)[0x4390f4]
/lib64/libpthread.so.0(+0x7aa1)[0x7fb562e89aa1]
/lib64/libc.so.6(clone+0x6d)[0x7fb5617c3aad]
2016-11-09 11:33:03 main.cpp:639:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2016-11-09 11:33:03 [INFO] Angel process is waiting 1 seconds before starting a new ProxySQL process
2016-11-09 11:33:04 [INFO] Angel process started ProxySQL process 12260
2016-11-09 11:33:04 ProxySQL_Admin.cpp:2870:flush_mysql_variables___database_to_runtime(): [ERROR] Impossible to set not existing variable session_debug with value "(null)". Deleting
2016-11-09 11:33:04 ProxySQL_Admin.cpp:2870:flush_mysql_variables___database_to_runtime(): [ERROR] Impossible to set not existing variable ping_interval_server with value "120000". Deleting
Standard ProxySQL Admin rev. 0.2.0902 -- ProxySQL_Admin.cpp -- Sat Oct 29 13:33:50 2016
Standard MySQL Threads Handler rev. 0.2.0902 -- MySQL_Thread.cpp -- Sat Oct 29 13:33:50 2016
Standard MySQL Authentication rev. 0.2.0902 -- MySQL_Authentication.cpp -- Sat Oct 29 13:33:50 2016
Standard Query Processor rev. 0.2.0902 -- Query_Processor.cpp -- Sat Oct 29 13:33:50 2016
In memory Standard Query Cache (SQC) rev. 1.2.0905 -- Query_Cache.cpp -- Sat Oct 29 13:33:50 2016
Standard MySQL Monitor (StdMyMon) rev. 1.2.0723 -- MySQL_Monitor.cpp -- Sat Oct 29 13:33:50 2016

[renecannao]

Hi Kenny,
Yes, seem the same.
Increasing priority on it.

[renecannao]

Fixed in 1.4.0 and 1.3.5