MariaDB connections exceeds personally-calculated ProxySQL max connections.

[ukyanj0]

Hello, I'd like to report an issue with the max connection mismatch bewtween ProxySQL and MariaDB when executing write queries.

The write DB server uses MariadDB(10.11.6). It is Galera-clustered with 1 write/4 read split configuration. The max_connections shows 5000. Hostgroup 0 deals with write quries.

The OS and ProxySQL versions are as below.

• ProxySQL version: ProxySQL version 2.3.2-10-g8cd66cf, codename Truls
• OS version: Rocky linux 8.8

We have 4 (non-ProxySQL-clustered) WAS servers. There are 8 processes that connect to a DB server running on each server and each process manages a mysql2 connection pool with 300 connect limit.

The ProxySQL variables and status.

MySQL [(none)]> select * from stats_mysql_connection_pool;
+-----------+---------------+----------+--------+----------+----------+--------+---------+-------------+----------+-------------------+-----------------+-----------------+------------+
| hostgroup | srv_host      | srv_port | status | ConnUsed | ConnFree | ConnOK | ConnERR | MaxConnUsed | Queries  | Queries_GTID_sync | Bytes_data_sent | Bytes_data_recv | Latency_us |
+-----------+---------------+----------+--------+----------+----------+--------+---------+-------------+----------+-------------------+-----------------+-----------------+------------+
| 1         | 192.168.xx.yy | 3306     | ONLINE | 0        | 100      | 701    | 0       | 147         | 17206284 | 0                 | 10834594831     | 43259720344     | 216        |
| 1         | 192.168.xx.yy | 3306     | ONLINE | 0        | 100      | 830    | 0       | 163         | 17188930 | 0                 | 10827462301     | 43358186552     | 285        |
| 1         | 192.168.xx.yy | 3306     | ONLINE | 0        | 65       | 826    | 15      | 152         | 17167823 | 0                 | 10821287628     | 43487815343     | 209        |
| 1         | 192.168.xx.yy | 3306     | ONLINE | 0        | 57       | 1586   | 0       | 176         | 17207175 | 0                 | 10832969577     | 43281205389     | 215        |
| 0         | 192.168.xx.yy | 3306     | ONLINE | 32       | 25       | 7386   | 0       | 909         | 12462426 | 0                 | 3163174498      | 293593159       | 321        |
+-----------+---------------+----------+--------+----------+----------+--------+---------+-------------+----------+-------------------+-----------------+-----------------+------------+

MySQL [(none)]> select * from stats_mysql_global;
+----------------------------------------------+----------------+
| Variable_Name                                | Variable_Value |
+----------------------------------------------+----------------+
| ProxySQL_Uptime                              | 1258073        |
| Active_Transactions                          | 0              |
| Client_Connections_aborted                   | 1              |
| Client_Connections_connected                 | 680            |
| Client_Connections_created                   | 1665564        |
| Server_Connections_aborted                   | 15             |
| Server_Connections_connected                 | 378            |
| Server_Connections_created                   | 11336          |
| Server_Connections_delayed                   | 0              |
| Client_Connections_non_idle                  | 41             |
| mysql_backend_buffers_bytes                  | 2765256        |
| mysql_frontend_buffers_bytes                 | 44564480       |
| mysql_session_internal_bytes                 | 736640         |
| Com_autocommit                               | 0              |
| Com_autocommit_filtered                      | 0              |
| Com_commit                                   | 6062           |
| Com_commit_filtered                          | 6062           |
| Com_rollback                                 | 0              |
| Com_rollback_filtered                        | 0              |
| Com_backend_change_user                      | 411333         |
| Com_backend_init_db                          | 1573506        |
| Com_backend_set_names                        | 449426         |
| Com_frontend_init_db                         | 0              |
| Com_frontend_set_names                       | 0              |
| Com_frontend_use_db                          | 3253020        |
| Com_backend_stmt_prepare                     | 8649430        |
| Com_backend_stmt_execute                     | 9077354        |
| Com_backend_stmt_close                       | 0              |
| Com_frontend_stmt_prepare                    | 9077356        |
| Com_frontend_stmt_execute                    | 9077356        |
| Com_frontend_stmt_close                      | 9077354        |
| Questions                                    | 94007954       |
| Slow_queries                                 | 1421           |
| GTID_consistent_queries                      | 0              |
| GTID_session_collected                       | 0              |
| Queries_backends_bytes_recv                  | 173708129694   |
| Queries_backends_bytes_sent                  | 46483731960    |
| Queries_frontends_bytes_recv                 | 47258381074    |
| Queries_frontends_bytes_sent                 | 233596857395   |
| Query_Processor_time_nsec                    | 0              |
| Backend_query_time_nsec                      | 0              |
| ConnPool_get_conn_latency_awareness          | 0              |
| ConnPool_get_conn_immediate                  | 3625487        |
| ConnPool_get_conn_success                    | 71225239       |
| ConnPool_get_conn_failure                    | 0              |
| mysql_killed_backend_connections             | 0              |
| mysql_killed_backend_queries                 | 0              |
| hostgroup_locked_set_cmds                    | 0              |
| hostgroup_locked_queries                     | 0              |
| mysql_unexpected_frontend_com_quit           | 0              |
| mysql_unexpected_frontend_packets            | 0              |
| queries_with_max_lag_ms__total_wait_time_us  | 0              |
| queries_with_max_lag_ms__delayed             | 0              |
| queries_with_max_lag_ms                      | 0              |
| backend_lagging_during_query                 | 0              |
| backend_offline_during_query                 | 0              |
| get_aws_aurora_replicas_skipped_during_query | 0              |
| automatic_detected_sql_injection             | 0              |
| whitelisted_sqli_fingerprint                 | 0              |
| max_connect_timeouts                         | 0              |
| generated_error_packets                      | 0              |
| client_host_error_killed_connections         | 0              |
| Client_Connections_hostgroup_locked          | 0              |
| Mirror_concurrency                           | 0              |
| Mirror_queue_length                          | 0              |
| Selects_for_update__autocommit0              | 0              |
| Servers_table_version                        | 1              |
| MySQL_Thread_Workers                         | 4              |
| Access_Denied_Wrong_Password                 | 1              |
| Access_Denied_Max_Connections                | 0              |
| Access_Denied_Max_User_Connections           | 0              |
| MySQL_Monitor_Workers                        | 2              |
| MySQL_Monitor_Workers_Aux                    | 0              |
| MySQL_Monitor_Workers_Started                | 2              |
| MySQL_Monitor_connect_check_OK               | 3143587        |
| MySQL_Monitor_connect_check_ERR              | 6              |
| MySQL_Monitor_ping_check_OK                  | 3143588        |
| MySQL_Monitor_ping_check_ERR                 | 1427           |
| MySQL_Monitor_read_only_check_OK             | 0              |
| MySQL_Monitor_read_only_check_ERR            | 0              |
| MySQL_Monitor_replication_lag_check_OK       | 0              |
| MySQL_Monitor_replication_lag_check_ERR      | 0              |
| MyHGM_myconnpoll_get                         | 67599752       |
| MyHGM_myconnpoll_get_ok                      | 67599752       |
| MyHGM_myconnpoll_push                        | 69348550       |
| MyHGM_myconnpoll_destroy                     | 7131           |
| MyHGM_myconnpoll_reset                       | 34000          |
| SQLite3_memory_bytes                         | 5557416        |
| ConnPool_memory_bytes                        | 33048696       |
| Stmt_Client_Active_Total                     | 0              |
| Stmt_Client_Active_Unique                    | 0              |
| Stmt_Server_Active_Total                     | 63             |
| Stmt_Server_Active_Unique                    | 62             |
| Stmt_Max_Stmt_id                             | 859            |
| Stmt_Cached                                  | 858            |
| Query_Cache_Memory_bytes                     | 0              |
| Query_Cache_count_GET                        | 0              |
| Query_Cache_count_GET_OK                     | 0              |
| Query_Cache_count_SET                        | 0              |
| Query_Cache_bytes_IN                         | 0              |
| Query_Cache_bytes_OUT                        | 0              |
| Query_Cache_Purged                           | 0              |
| Query_Cache_Entries                          | 0              |
| new_req_conns_count                          | 0              |
+----------------------------------------------+----------------+

MySQL [(none)]> select * from mysql_servers;
+--------------+---------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname      | port | gtid_port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+---------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 1            | 192.168.xx.yy | 3306 | 0         | ONLINE | 1      | 0           | 1000            | 0                   | 0       | 0              |         |
| 0            | 192.168.xx.yy | 3306 | 0         | ONLINE | 1      | 0           | 1000            | 0                   | 0       | 0              |         |
| 1            | 192.168.xx.yy | 3306 | 0         | ONLINE | 1      | 0           | 1000            | 0                   | 0       | 0              |         |
| 1            | 192.168.xx.yy | 3306 | 0         | ONLINE | 1      | 0           | 1000            | 0                   | 0       | 0              |         |
| 1            | 192.168.xx.yy | 3306 | 0         | ONLINE | 1      | 0           | 1000            | 0                   | 0       | 0              |         |
+--------------+---------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+

MySQL [(none)]> select * from global_variables;
+----------------------------------------------------------------------+---------------------------------+
| variable_name                                                        | variable_value                  |
+----------------------------------------------------------------------+---------------------------------+
| mysql-default_charset                                                | utf8                            |
| mysql-default_collation_connection                                   | utf8_general_ci                 |
| mysql-shun_on_failures                                               | 5                               |
| mysql-shun_recovery_time_sec                                         | 10                              |
| mysql-query_retries_on_failure                                       | 1                               |
| mysql-client_multi_statements                                        | true                            |
| mysql-client_host_cache_size                                         | 0                               |
| mysql-client_host_error_counts                                       | 0                               |
| mysql-connect_retries_delay                                          | 1                               |
| mysql-connection_delay_multiplex_ms                                  | 0                               |
| mysql-connection_max_age_ms                                          | 0                               |
| mysql-connect_timeout_client                                         | 10000                           |
| mysql-connect_timeout_server_max                                     | 10000                           |
| mysql-enable_client_deprecate_eof                                    | true                            |
| mysql-enable_server_deprecate_eof                                    | true                            |
| mysql-enable_load_data_local_infile                                  | false                           |
| mysql-eventslog_filename                                             |                                 |
| mysql-eventslog_filesize                                             | 104857600                       |
| mysql-eventslog_default_log                                          | 0                               |
| mysql-eventslog_format                                               | 1                               |
| mysql-auditlog_filename                                              |                                 |
| mysql-auditlog_filesize                                              | 104857600                       |
| mysql-handle_unknown_charset                                         | 1                               |
| mysql-free_connections_pct                                           | 10                              |
| mysql-connection_warming                                             | false                           |
| mysql-session_idle_ms                                                | 1                               |
| mysql-have_ssl                                                       | false                           |
| mysql-client_found_rows                                              | true                            |
| mysql-log_mysql_warnings_enabled                                     | false                           |
| mysql-monitor_enabled                                                | true                            |
| mysql-monitor_connect_timeout                                        | 600                             |
| mysql-monitor_ping_max_failures                                      | 3                               |
| mysql-monitor_ping_timeout                                           | 1000                            |
| mysql-monitor_read_only_max_timeout_count                            | 3                               |
| mysql-monitor_replication_lag_interval                               | 10000                           |
| mysql-monitor_replication_lag_timeout                                | 1000                            |
| mysql-monitor_replication_lag_count                                  | 1                               |
| mysql-monitor_groupreplication_healthcheck_interval                  | 5000                            |
| mysql-monitor_groupreplication_healthcheck_timeout                   | 800                             |
| mysql-monitor_groupreplication_healthcheck_max_timeout_count         | 3                               |
| mysql-monitor_groupreplication_max_transactions_behind_count         | 3                               |
| mysql-monitor_groupreplication_max_transactions_behind_for_read_only | 1                               |
| mysql-monitor_galera_healthcheck_interval                            | 5000                            |
| mysql-monitor_galera_healthcheck_timeout                             | 800                             |
| mysql-monitor_galera_healthcheck_max_timeout_count                   | 3                               |
| mysql-monitor_replication_lag_use_percona_heartbeat                  |                                 |
| mysql-monitor_query_interval                                         | 60000                           |
| mysql-monitor_query_timeout                                          | 100                             |
| mysql-monitor_slave_lag_when_null                                    | 60                              |
| mysql-monitor_threads_min                                            | 8                               |
| mysql-monitor_threads_max                                            | 128                             |
| mysql-monitor_threads_queue_maxsize                                  | 128                             |
| mysql-monitor_wait_timeout                                           | true                            |
| mysql-monitor_writer_is_also_reader                                  | true                            |
| mysql-max_allowed_packet                                             | 67108864                        |
| mysql-tcp_keepalive_time                                             | 0                               |
| mysql-use_tcp_keepalive                                              | false                           |
| mysql-automatic_detect_sqli                                          | false                           |
| mysql-firewall_whitelist_enabled                                     | false                           |
| mysql-firewall_whitelist_errormsg                                    | Firewall blocked this query     |
| mysql-throttle_connections_per_sec_to_hostgroup                      | 1000000                         |
| mysql-max_transaction_idle_time                                      | 14400000                        |
| mysql-max_transaction_time                                           | 14400000                        |
| mysql-multiplexing                                                   | true                            |
| mysql-log_unhealthy_connections                                      | true                            |
| mysql-enforce_autocommit_on_reads                                    | false                           |
| mysql-autocommit_false_not_reusable                                  | false                           |
| mysql-autocommit_false_is_transaction                                | false                           |
| mysql-verbose_query_error                                            | false                           |
| mysql-hostgroup_manager_verbose                                      | 1                               |
| mysql-binlog_reader_connect_retry_msec                               | 3000                            |
| mysql-threshold_query_length                                         | 524288                          |
| mysql-threshold_resultset_size                                       | 4194304                         |
| mysql-query_digests_max_digest_length                                | 2048                            |
| mysql-query_digests_max_query_length                                 | 65000                           |
| mysql-query_digests_grouping_limit                                   | 3                               |
| mysql-wait_timeout                                                   | 28800000                        |
| mysql-throttle_max_bytes_per_second_to_client                        | 0                               |
| mysql-throttle_ratio_server_to_client                                | 0                               |
| mysql-max_stmts_per_connection                                       | 20                              |
| mysql-max_stmts_cache                                                | 10000                           |
| mysql-mirror_max_concurrency                                         | 16                              |
| mysql-mirror_max_queue_length                                        | 32000                           |
| mysql-default_max_latency_ms                                         | 1000                            |
| mysql-query_processor_iterations                                     | 0                               |
| mysql-query_processor_regex                                          | 1                               |
| mysql-set_query_lock_on_hostgroup                                    | 1                               |
| mysql-reset_connection_algorithm                                     | 2                               |
| mysql-auto_increment_delay_multiplex                                 | 5                               |
| mysql-long_query_time                                                | 1000                            |
| mysql-query_cache_size_MB                                            | 256                             |
| mysql-poll_timeout_on_failure                                        | 100                             |
| mysql-server_capabilities                                            | 569867                          |
| mysql-keep_multiplexing_variables                                    | tx_isolation,version            |
| mysql-kill_backend_connection_when_disconnect                        | true                            |
| mysql-client_session_track_gtid                                      | true                            |
| mysql-session_idle_show_processlist                                  | true                            |
| mysql-show_processlist_extended                                      | 0                               |
| mysql-query_digests                                                  | true                            |
| mysql-query_digests_lowercase                                        | false                           |
| mysql-query_digests_replace_null                                     | false                           |
| mysql-query_digests_no_digits                                        | false                           |
| mysql-query_digests_normalize_digest_text                            | false                           |
| mysql-query_digests_track_hostname                                   | false                           |
| mysql-servers_stats                                                  | true                            |
| mysql-default_reconnect                                              | true                            |
| mysql-ssl_p2s_ca                                                     |                                 |
| mysql-ssl_p2s_capath                                                 |                                 |
| mysql-ssl_p2s_cert                                                   |                                 |
| mysql-ssl_p2s_key                                                    |                                 |
| mysql-ssl_p2s_cipher                                                 |                                 |
| mysql-ssl_p2s_crl                                                    |                                 |
| mysql-ssl_p2s_crlpath                                                |                                 |
| mysql-init_connect                                                   |                                 |
| mysql-ldap_user_variable                                             |                                 |
| mysql-add_ldap_user_comment                                          |                                 |
| mysql-default_tx_isolation                                           | READ-COMMITTED                  |
| mysql-default_session_track_gtids                                    | OFF                             |
| mysql-connpoll_reset_queue_length                                    | 50                              |
| mysql-min_num_servers_lantency_awareness                             | 1000                            |
| mysql-aurora_max_lag_ms_only_read_from_replicas                      | 2                               |
| mysql-stats_time_backend_query                                       | false                           |
| mysql-stats_time_query_processor                                     | false                           |
| mysql-query_cache_stores_empty_result                                | true                            |
| admin-stats_credentials                                              | stats:stats                     |
| admin-stats_mysql_connections                                        | 60                              |
| admin-stats_mysql_connection_pool                                    | 60                              |
| admin-stats_mysql_query_cache                                        | 60                              |
| admin-stats_mysql_query_digest_to_disk                               | 0                               |
| admin-stats_system_cpu                                               | 60                              |
| admin-stats_system_memory                                            | 60                              |
| admin-telnet_admin_ifaces                                            | (null)                          |
| admin-telnet_stats_ifaces                                            | (null)                          |
| admin-read_only                                                      | false                           |
| admin-hash_passwords                                                 | true                            |
| admin-vacuum_stats                                                   | true                            |
| admin-version                                                        | 2.3.2-10-g8cd66cf               |
| admin-cluster_username                                               |                                 |
| admin-cluster_password                                               |                                 |
| admin-cluster_check_interval_ms                                      | 1000                            |
| admin-cluster_check_status_frequency                                 | 10                              |
| admin-cluster_mysql_query_rules_diffs_before_sync                    | 3                               |
| admin-cluster_mysql_servers_diffs_before_sync                        | 3                               |
| admin-cluster_mysql_users_diffs_before_sync                          | 3                               |
| admin-cluster_proxysql_servers_diffs_before_sync                     | 3                               |
| admin-cluster_mysql_variables_diffs_before_sync                      | 3                               |
| admin-cluster_admin_variables_diffs_before_sync                      | 3                               |
| admin-cluster_ldap_variables_diffs_before_sync                       | 3                               |
| admin-cluster_mysql_query_rules_save_to_disk                         | true                            |
| admin-cluster_mysql_servers_save_to_disk                             | true                            |
| admin-cluster_mysql_users_save_to_disk                               | true                            |
| admin-cluster_proxysql_servers_save_to_disk                          | true                            |
| admin-cluster_mysql_variables_save_to_disk                           | true                            |
| admin-cluster_admin_variables_save_to_disk                           | true                            |
| admin-cluster_ldap_variables_save_to_disk                            | true                            |
| admin-checksum_mysql_query_rules                                     | true                            |
| admin-checksum_mysql_servers                                         | true                            |
| admin-checksum_mysql_users                                           | true                            |
| admin-checksum_mysql_variables                                       | true                            |
| admin-checksum_admin_variables                                       | true                            |
| admin-checksum_ldap_variables                                        | true                            |
| admin-restapi_enabled                                                | false                           |
| admin-restapi_port                                                   | 6070                            |
| admin-web_enabled                                                    | false                           |
| admin-web_port                                                       | 6080                            |
| admin-web_verbosity                                                  | 0                               |
| admin-prometheus_memory_metrics_interval                             | 61                              |
| admin-admin_credentials                                              | admin:admin                     |
| admin-mysql_ifaces                                                   | 0.0.0.0:6032                    |
| admin-refresh_interval                                               | 2000                            |
| mysql-threads                                                        | 4                               |
| mysql-max_connections                                                | 2048                            |
| mysql-default_query_delay                                            | 0                               |
| mysql-default_query_timeout                                          | 36000000                        |
| mysql-have_compress                                                  | true                            |
| mysql-poll_timeout                                                   | 2000                            |
| mysql-interfaces                                                     | 0.0.0.0:6033;/tmp/proxysql.sock |
| mysql-default_schema                                                 | information_schema              |
| mysql-stacksize                                                      | 1048576                         |
| mysql-server_version                                                 | 5.5.30                          |
| mysql-connect_timeout_server                                         | 3000                            |
| mysql-monitor_connect_interval                                       | 2000                            |
| mysql-monitor_ping_interval                                          | 2000                            |
| mysql-ping_interval_server_msec                                      | 120000                          |
| mysql-ping_timeout_server                                            | 500                             |
| mysql-commands_stats                                                 | true                            |
| mysql-sessions_sort                                                  | true                            |
| mysql-monitor_username                                               | monitor                         |
| mysql-monitor_password                                               | monitor                         |
| mysql-monitor_history                                                | 600000                          |
| mysql-monitor_read_only_interval                                     | 2000                            |
| mysql-monitor_read_only_timeout                                      | 500                             |
| mysql-connect_retries_on_failure                                     | 10                              |
+----------------------------------------------------------------------+---------------------------------+

mysql-max_connections | 2048 from "select * from global_variables;"
MaxConnUsed | 909 from "select * from stats_mysql_connection_pool;"
max_connections | 1000 from from "select * from mysql_servers;"

I personally expected that with 1000 max_connections and 4 WAS servers, the maximum connection for the write DB server(hostgroup: 0) won't exceed 4000. However, I could monitor more than 4000 thread_connected out of of 5000 max connections on the write DB server. The query was like

INSERT INTO TABLE1 (A,B,C) 
SELECT A,B,C FROM TABLE2
LIMIT 0,500;

INSERT INTO TABLE1 (A,B,C) 
SELECT A,B,C FROM TABLE2
LIMIT 500,500;

INSERT INTO TABLE1 (A,B,C) 
SELECT A,B,C FROM TABLE2
LIMIT 1000,500;

...

INSERT INTO TABLE1 (A,B,C) 
SELECT A,B,C FROM TABLE2
LIMIT 30000,500;

• The first guess I could make was that 2048 mysql-max_connections variable overode 1000 max_connections configuration and now it allows each WAS server has up to 2048 connection.
• Second one is that each of 4 WAS has around 900 MaxConnUsed for hostgroup 0 that is summed up to ~3600. (which makes me think the 1000 max_connection configuration worked properly) However, the looping query(LIMIT X,500) forces creating connections, so it was not able to reuse a connection somehow even though multiplex option was true. The connection seemed to exceed 4000 "only temporarily" before the garbage connection was broken.

I always appreciate your support.

Sincerely
Luke