Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ProxySQL 2.5.5 crashes on stats_mysql_query_digest query #4399

Closed
TokuiNico opened this issue Dec 15, 2023 · 6 comments · Fixed by #4414
Closed

ProxySQL 2.5.5 crashes on stats_mysql_query_digest query #4399

TokuiNico opened this issue Dec 15, 2023 · 6 comments · Fixed by #4414
Assignees
@rahim-kanji

Comments

@TokuiNico
Copy link

TokuiNico commented Dec 15, 2023 •
edited
Loading

Docker Image

proxysql crashes after query stats_mysql_query_digest quickly or query stats_mysql_query_digest after quering stats_mysql_query_digest_rest somtimes

SQL query

select count(*) from stats_mysql_query_digest;

LOG

Dec 15 11:19:02 SERVER docker[65225]: proxysql: Query_Processor.cpp:252: char* QP_query_digest_stats::get_digest_text(const umap_query_digest_text*): Assertion `0' failed.
Dec 15 11:19:02 SERVER docker[65225]: Error: signal 6:
Dec 15 11:19:02 SERVER docker[65225]: proxysql(_Z13crash_handleri+0x29)[0x559afacb5cd9]
Dec 15 11:19:02 SERVER docker[65225]: /lib/x86_64-linux-gnu/libc.so.6(+0x38d60)[0x7f67331e3d60]
Dec 15 11:19:02 SERVER docker[65225]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x141)[0x7f67331e3ce1]
Dec 15 11:19:02 SERVER docker[65225]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x123)[0x7f67331cd537]
Dec 15 11:19:02 SERVER docker[65225]: /lib/x86_64-linux-gnu/libc.so.6(+0x2240f)[0x7f67331cd40f]
Dec 15 11:19:02 SERVER docker[65225]: /lib/x86_64-linux-gnu/libc.so.6(+0x31662)[0x7f67331dc662]
Dec 15 11:19:02 SERVER docker[65225]: proxysql(+0x37c1e3)[0x559afad571e3]
Dec 15 11:19:02 SERVER docker[65225]: proxysql(_ZN14ProxySQL_Admin41stats___save_mysql_query_digest_to_sqliteEbbPK14SQLite3_resultPKSt13unordered_mapImPvSt4hashImESt8equal_toImESaISt4pairIKmS4_EEEPKS3_ImPcS6_S8_SaIS9_ISA_SG_EEE+0x1713)[0x559afad99ea3]
Dec 15 11:19:02 SERVER docker[65225]: proxysql(_ZN15Query_Processor20get_query_digests_v2Eb+0xa8f)[0x559afad5ec7f]
Dec 15 11:19:02 SERVER docker[65225]: proxysql(_ZN14ProxySQL_Admin30stats___mysql_query_digests_v2Ebbb+0x2c)[0x559afad9b6dc]
Dec 15 11:19:02 SERVER docker[65225]: proxysql(_ZN14ProxySQL_Admin24GenericRefreshStatisticsEPKcjb+0xade)[0x559afada3f0e]
Dec 15 11:19:02 SERVER docker[65225]: proxysql(_Z21admin_session_handlerP13MySQL_SessionPvP10_PtrSize_t+0x1aa)[0x559afadb0dda]
Dec 15 11:19:02 SERVER docker[65225]: proxysql(_ZN13MySQL_Session80handler___status_WAITING_CLIENT_DATA___STATE_SLEEP___MYSQL_COM_QUERY___not_mysqlER10_PtrSize_t+0x2c)[0x559afad2c62c]
Dec 15 11:19:02 SERVER docker[65225]: proxysql(_ZN13MySQL_Session20get_pkts_from_clientERbR10_PtrSize_t+0xec6)[0x559afad44176]
Dec 15 11:19:02 SERVER docker[65225]: proxysql(_ZN13MySQL_Session7handlerEv+0xce)[0x559afad44c9e]
Dec 15 11:19:02 SERVER docker[65225]: proxysql(_Z11child_mysqlPv+0x3ae)[0x559afad6573e]
Dec 15 11:19:02 SERVER docker[65225]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7)[0x7f67336c2ea7]
Dec 15 11:19:02 SERVER docker[65225]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f67332a6a2f]
Dec 15 11:19:02 SERVER docker[65225]:  ---- proxysql(_Z13crash_handleri+0x29) [0x559afacb5cd9] : crash_handler(int)
Dec 15 11:19:02 SERVER docker[65225]:  ---- proxysql(_ZN15Query_Processor20get_query_digests_v2Eb+0xa8f) [0x559afad5ec7f] : Query_Processor::get_query_digests_v2(bool)
Dec 15 11:19:02 SERVER docker[65225]:  ---- proxysql(_ZN14ProxySQL_Admin30stats___mysql_query_digests_v2Ebbb+0x2c) [0x559afad9b6dc] : ProxySQL_Admin::stats___mysql_query_digests_v2(bool, bool, bool)
Dec 15 11:19:02 SERVER docker[65225]:  ---- proxysql(_ZN14ProxySQL_Admin24GenericRefreshStatisticsEPKcjb+0xade) [0x559afada3f0e] : ProxySQL_Admin::GenericRefreshStatistics(char const*, unsigned int, bool)
Dec 15 11:19:02 SERVER docker[65225]:  ---- proxysql(_Z21admin_session_handlerP13MySQL_SessionPvP10_PtrSize_t+0x1aa) [0x559afadb0dda] : admin_session_handler(MySQL_Session*, void*, _PtrSize_t*)
Dec 15 11:19:02 SERVER docker[65225]:  ---- proxysql(_ZN13MySQL_Session20get_pkts_from_clientERbR10_PtrSize_t+0xec6) [0x559afad44176] : MySQL_Session::get_pkts_from_client(bool&, _PtrSize_t&)
Dec 15 11:19:02 SERVER docker[65225]:  ---- proxysql(_ZN13MySQL_Session7handlerEv+0xce) [0x559afad44c9e] : MySQL_Session::handler()
Dec 15 11:19:02 SERVER docker[65225]:  ---- proxysql(_Z11child_mysqlPv+0x3ae) [0x559afad6573e] : child_mysql(void*)
Dec 15 11:19:02 SERVER docker[65225]: To report a crashing bug visit: https://github.com/sysown/proxysql/issues
Dec 15 11:19:02 SERVER docker[65225]: For support visit: https://proxysql.com/services/support/
Dec 15 11:19:02 SERVER docker[65225]: Error: signal 11:
Dec 15 11:19:02 SERVER docker[65225]: proxysql(_Z13crash_handleri+0x29)[0x559afacb5cd9]
Dec 15 11:19:02 SERVER docker[65225]: /lib/x86_64-linux-gnu/libc.so.6(+0x38d60)[0x7f67331e3d60]
Dec 15 11:19:02 SERVER docker[65225]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x1ee)[0x7f67331cd602]
Dec 15 11:19:02 SERVER docker[65225]: /lib/x86_64-linux-gnu/libc.so.6(+0x2240f)[0x7f67331cd40f]
Dec 15 11:19:02 SERVER docker[65225]: /lib/x86_64-linux-gnu/libc.so.6(+0x31662)[0x7f67331dc662]
Dec 15 11:19:02 SERVER docker[65225]: proxysql(+0x37c1e3)[0x559afad571e3]
Dec 15 11:19:02 SERVER docker[65225]: proxysql(_ZN14ProxySQL_Admin41stats___save_mysql_query_digest_to_sqliteEbbPK14SQLite3_resultPKSt13unordered_mapImPvSt4hashImESt8equal_toImESaISt4pairIKmS4_EEEPKS3_ImPcS6_S8_SaIS9_ISA_SG_EEE+0x1713)[0x559afad99ea3]
Dec 15 11:19:02 SERVER docker[65225]: proxysql(_ZN15Query_Processor20get_query_digests_v2Eb+0xa8f)[0x559afad5ec7f]
Dec 15 11:19:02 SERVER docker[65225]: proxysql(_ZN14ProxySQL_Admin30stats___mysql_query_digests_v2Ebbb+0x2c)[0x559afad9b6dc]
Dec 15 11:19:02 SERVER docker[65225]: proxysql(_ZN14ProxySQL_Admin24GenericRefreshStatisticsEPKcjb+0xade)[0x559afada3f0e]
Dec 15 11:19:02 SERVER docker[65225]: proxysql(_Z21admin_session_handlerP13MySQL_SessionPvP10_PtrSize_t+0x1aa)[0x559afadb0dda]
Dec 15 11:19:02 SERVER docker[65225]: proxysql(_ZN13MySQL_Session80handler___status_WAITING_CLIENT_DATA___STATE_SLEEP___MYSQL_COM_QUERY___not_mysqlER10_PtrSize_t+0x2c)[0x559afad2c62c]
Dec 15 11:19:02 SERVER docker[65225]: proxysql(_ZN13MySQL_Session20get_pkts_from_clientERbR10_PtrSize_t+0xec6)[0x559afad44176]
Dec 15 11:19:02 SERVER docker[65225]: proxysql(_ZN13MySQL_Session7handlerEv+0xce)[0x559afad44c9e]
Dec 15 11:19:02 SERVER docker[65225]: proxysql(_Z11child_mysqlPv+0x3ae)[0x559afad6573e]
Dec 15 11:19:02 SERVER docker[65225]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7)[0x7f67336c2ea7]
Dec 15 11:19:02 SERVER docker[65225]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f67332a6a2f]
Dec 15 11:19:02 SERVER docker[65225]:  ---- proxysql(_Z13crash_handleri+0x29) [0x559afacb5cd9] : crash_handler(int)
Dec 15 11:19:02 SERVER docker[65225]:  ---- proxysql(_ZN15Query_Processor20get_query_digests_v2Eb+0xa8f) [0x559afad5ec7f] : Query_Processor::get_query_digests_v2(bool)
Dec 15 11:19:02 SERVER docker[65225]:  ---- proxysql(_ZN14ProxySQL_Admin30stats___mysql_query_digests_v2Ebbb+0x2c) [0x559afad9b6dc] : ProxySQL_Admin::stats___mysql_query_digests_v2(bool, bool, bool)
Dec 15 11:19:02 SERVER docker[65225]:  ---- proxysql(_ZN14ProxySQL_Admin24GenericRefreshStatisticsEPKcjb+0xade) [0x559afada3f0e] : ProxySQL_Admin::GenericRefreshStatistics(char const*, unsigned int, bool)
Dec 15 11:19:02 SERVER docker[65225]:  ---- proxysql(_Z21admin_session_handlerP13MySQL_SessionPvP10_PtrSize_t+0x1aa) [0x559afadb0dda] : admin_session_handler(MySQL_Session*, void*, _PtrSize_t*)
Dec 15 11:19:02 SERVER docker[65225]:  ---- proxysql(_ZN13MySQL_Session20get_pkts_from_clientERbR10_PtrSize_t+0xec6) [0x559afad44176] : MySQL_Session::get_pkts_from_client(bool&, _PtrSize_t&)
Dec 15 11:19:02 SERVER docker[65225]:  ---- proxysql(_ZN13MySQL_Session7handlerEv+0xce) [0x559afad44c9e] : MySQL_Session::handler()
Dec 15 11:19:02 SERVER docker[65225]:  ---- proxysql(_Z11child_mysqlPv+0x3ae) [0x559afad6573e] : child_mysql(void*)
Dec 15 11:19:02 SERVER docker[65225]: To report a crashing bug visit: https://github.com/sysown/proxysql/issues
Dec 15 11:19:02 SERVER docker[65225]: For support visit: https://proxysql.com/services/support/```
@renecannao
Copy link
Contributor

Hi. Do you have a core dump available?

@renecannao
Copy link
Contributor

2nd question: all crashes report this?

Query_Processor.cpp:252: char* QP_query_digest_stats::get_digest_text(const umap_query_digest_text*): Assertion `0' failed.

@TokuiNico
Copy link
Author

Hi @renecannao
I didn't find any core dumps logs. Should I enable some settings?

I tried several times the error msg appears every time when query select count(*) from stats_mysql_query_digest; quickly (3 ~ 5 times in one second). And it happened easily in PRODUCTION environment, even just query 2 times.

Query_Processor.cpp:252: char* QP_query_digest_stats::get_digest_text(const umap_query_digest_text*): Assertion `0' failed.

There are configs related to digest

| variable_name                             | variable_value |
|-------------------------------------------|----------------|
| mysql-query_digests_max_digest_length     | 2048           |
| mysql-query_digests_max_query_length      | 65000          |
| mysql-query_digests_grouping_limit        | 1              |
| mysql-query_digests_groups_grouping_limit | 0              |
| mysql-query_digests                       | true           |
| mysql-query_digests_lowercase             | 1              |
| mysql-query_digests_replace_null          | 1              |
| mysql-query_digests_no_digits             | false          |
| mysql-query_digests_normalize_digest_text | 1              |
| mysql-query_digests_track_hostname        | false          |
| mysql-query_digests_keep_comment          | false          |
| mysql-parse_failure_logs_digest           | false          |
| admin-stats_mysql_query_digest_to_disk    | 3600           |

@renecannao
Copy link
Contributor

For now please disable mysql-query_digests_normalize_digest_text and see if the crashes stop.
We will try to investigate this further.

Thanks

@TokuiNico
Copy link
Author

I have disabled mysql-query_digests_normalize_digest_text and after retesting, the server indeed did not crash. Thank you for your assistance!

@renecannao
Copy link
Contributor

Good it stopped crashing.
Yet, I am reopening this issue: it is a crashing bug and it needs further investigations

@renecannao renecannao reopened this Dec 15, 2023
@rahim-kanji rahim-kanji self-assigned this Jan 10, 2024
@rahim-kanji rahim-kanji mentioned this issue Jan 11, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rahim-kanji rahim-kanji

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fixed crash on frequently querying stats_mysql_query_digest* tables sysown/proxysql
3 participants
@renecannao @TokuiNico @rahim-kanji