Crash on mysql_connection.cpp:417:set_names_start()

[riveraja]

Standard MySQL Monitor (StdMyMon) rev. 1.2.0723 -- MySQL_Monitor.cpp -- Wed Sep 28 21:48:11 2016
2016-11-30 09:14:18 mysql_connection.cpp:417:set_names_start(): [ERROR] Not existing charset number 0
proxysql: mysql_connection.cpp:418: void MySQL_Connection::set_names_start(): Assertion `0' failed.
Error: signal 6:
/usr/bin/proxysql(_Z13crash_handleri+0x1a)[0x4386da]
/lib64/libc.so.6(+0x32660)[0x7f8882ec3660]
/lib64/libc.so.6(gsignal+0x35)[0x7f8882ec35e5]
/lib64/libc.so.6(abort+0x175)[0x7f8882ec4dc5]
/lib64/libc.so.6(+0x2b70e)[0x7f8882ebc70e]
/lib64/libc.so.6(__assert_perror_fail+0x0)[0x7f8882ebc7d0]
/usr/bin/proxysql[0x4fca14]
/usr/bin/proxysql(_ZN16MySQL_Connection7handlerEs+0x4d0)[0x4fd0a0]
/usr/bin/proxysql(_ZN16MySQL_Connection15async_set_namesEsh+0x48)[0x4fdd68]
/usr/bin/proxysql(_ZN13MySQL_Session7handlerEv+0xf76)[0x45c2a6]
/usr/bin/proxysql(_ZN12MySQL_Thread20process_all_sessionsEv+0x1b2)[0x4520e2]
/usr/bin/proxysql(_ZN12MySQL_Thread3runEv+0x54b)[0x4543eb]
/usr/bin/proxysql(_Z24mysql_worker_thread_funcPv+0x54)[0x437074]
/lib64/libpthread.so.0(+0x7aa1)[0x7f888463faa1]
/lib64/libc.so.6(clone+0x6d)[0x7f8882f79aad]
2016-11-30 09:14:18 main.cpp:608:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2016-11-30 09:14:18 [INFO] Angel process is waiting 1 seconds before starting a new ProxySQL process
2016-11-30 09:14:19 [INFO] Angel process started ProxySQL process 583

When starting nodejs application ghost (a blogging platform) with charset: utf8, application proxysql will crash with above stacktrace.

After commenting out charset option, app started fine:

        database: {
            client: 'mysql',
            connection: {
                host: '10.0.3.49',
		port: '6033',
		user: 'ghostusr',
		password: 'ghostpwd1',
		database: 'ghostdb' //,
		// charset: 'utf8'   <== when commented out app will start
            },
	    pool: {
		min: 2,
		max: 250
	}

mysql-default_charset is utf8 in global_variables table

[renecannao]

Jericho,
Thank you for the report.
Just before the crash (due to an assert()) I can see:

mysql_connection.cpp:417:set_names_start(): [ERROR] Not existing charset number 0

Either the client is really asking a non existing charset (unlikely, seems a plain utf8), or ProxySQL is parsing the request incorrectly (this is probably the case).

To be honest I never executed any nodejs application, and perhaps setting one will take some time.
To debug the issue, could you please log some network traffic when the issue happens, so I can check what request the client is sending?
For example, in the server where proxysql is running, you can execute:

tcpdump -s 0 -w issue810.tcp port 6033

Then try to connect using the client, and finally compress and attach the file to this issue.
Thank you.

[riveraja]

Hi Rene,

Here's the tcpdump file, I hope this helps.
issue810.tcp.zip

[renecannao]

Jericho,
the output is really interesting.
From tcpdump, there are some of the packets sent from the clients during the authentication:

        0x0030:            4600 0001 cff3 0200 0000 0000  .I.RF...........
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0050:  0000 0000 0000 0000 6768 6f73 7475 7372  ........ghostusr
        0x0060:  0014 bbbe 8693 d922 f4ba ce4f 2abf 3a81  ......."...O*.:.
        0x0070:  dc1b 53f0 602a 6768 6f73 7464 6200       ..S.`*ghostdb.

        0x0030:            4600 0001 cff3 0200 0000 0000  .I.SF...........
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0050:  0000 0000 0000 0000 6768 6f73 7475 7372  ........ghostusr
        0x0060:  0014 240c d928 c444 8210 776d 1565 cb50  ..$..(.D..wm.e.P
        0x0070:  e5f6 40cb 37ee 6768 6f73 7464 6200       [email protected].

        0x0030:            4600 0001 cff3 0200 0000 0000  .I.SF...........
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0050:  0000 0000 0000 0000 6768 6f73 7475 7372  ........ghostusr
        0x0060:  0014 3feb a246 0a59 a2a3 2a36 86bb 529b  ..?..F.Y..*6..R.
        0x0070:  168a 5e4b 7a76 6768 6f73 7464 6200       ..^Kzvghostdb.

In every packet, the first 4 bytes are a header: 4600 0001 .
What follows is the payload used during authentication.
According to the documentation, the payload starts with:

4              capability flags, CLIENT_PROTOCOL_41 always set
4              max-packet size
1              character set
string[23]     reserved (all [0])
string[NUL]    username

Breaking down the payload:

• cff3 0200 is capability flags
• 0000 0000 is max-packet size
• 00 is charset
• 00 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 are 23 bytes all 0
• 6768 6f73 7475 7372 00 is a null terminated username (ghostusr).

So, indeed, the driver you are using is specifying charset 0 .
Interestingly, it is specifying 0 also for max-packet size .

As I assume you are successfully able to connect directly to MySQL Server, I started looking into it.
From MySQL Server source code:

Use server character set and collation if

• opt_character_set_client_handshake is not set
• client has not specified a character set
• client character set is the same as the servers
• client character set doesn't exists in server

I guess in this case applies "client has not specified a character set", because your driver is setting charset=0. I will patch ProxySQL to handle this case too.

Meanwhile, a possible workaround (please try it) is to set mysql-default_charset=latin1 in global_variables : this will probably force nodejs driver to specify a charset (utf8 is 33) and ProxySQL will apply it.

Thanks.

[renecannao]

I will consider this a bug, and therefore will be applied also to 1.2.x and 1.3.x releases.

[riveraja]

Rene,

The proposed workaround didn't work, I think node-mysql has an existing bug with the charset option. I added debug: true on the connection config and saw this when charset: 'utf8' was enabled:

--> ClientAuthenticationPacket
ClientAuthenticationPacket {
  clientFlags: 193487,
  maxPacketSize: 0,
  **charsetNumber: undefined,**
  filler: undefined,
  user: 'ghostusr',
  scrambleBuff: <Buffer 77 45 a2 ae d8 95 79 e3 49 14 2d 82 ad 6e d5 f6 8d 38 49 48>,
  database: 'ghostdb',
  protocol41: true }

But when disabled/commented out:

--> ClientAuthenticationPacket
ClientAuthenticationPacket {
  clientFlags: 193487,
  maxPacketSize: 0,
  **charsetNumber: 33,**
  filler: undefined,
  user: 'ghostusr',
  scrambleBuff: <Buffer 0e 0f 4b b4 f9 92 d4 21 53 43 23 a9 00 75 a6 7d a4 fd 47 ad>,
  database: 'ghostdb',
  protocol41: true }

And yes, this works fine when I connect the application directly to the mysql backend with or without charset: 'utf8' used.

[renecannao]

Applied in branches v1.2.6 , v1.3.0 and v1.4.0

[naughtyGitCat]

@renecannao i meet this bug again.proxysql:1.4.3 mysql:5.7.20

[naughtyGitCat]

proxysql.zip
@renecannao

it's the error dump file