Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix pxc_maint_mode failback when writer_is_also_reader=2 #2671

Merged
merged 1 commit into from
Apr 28, 2020
Merged

Fix pxc_maint_mode failback when writer_is_also_reader=2 #2671

merged 1 commit into from
Apr 28, 2020

Conversation

alpes214
Copy link
Contributor

Description:
This PR fxies issue #2656

Testing:

  1. Setup mysql_galera_hostgroups
MySQL [(none)]> select * from mysql_galera_hostgroups;
+------------------+-------------------------+------------------+-------------------+--------+-------------+-----------------------+-------------------------+---------+
| writer_hostgroup | backup_writer_hostgroup | reader_hostgroup | offline_hostgroup | active | max_writers | writer_is_also_reader | max_transactions_behind | comment |
+------------------+-------------------------+------------------+-------------------+--------+-------------+-----------------------+-------------------------+---------+
| 1                | 2                       | 3                | 4                 | 1      | 1           | 2                     | 150                     | NULL    |
+------------------+-------------------------+------------------+-------------------+--------+-------------+-----------------------+-------------------------+---------+
1 row in set (0.00 sec)

When server just started, three nodes are distributed like this:

MySQL [(none)]> select * from runtime_mysql_servers;
+--------------+-------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname    | port | gtid_port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+-------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 1            | 172.18.0.30 | 3306 | 0         | ONLINE | 1      | 0           | 1000            | 0                   | 0       | 0              |         |
| 2            | 172.18.0.20 | 3306 | 0         | ONLINE | 1      | 0           | 1000            | 0                   | 0       | 0              |         |
| 2            | 172.18.0.10 | 3306 | 0         | ONLINE | 1      | 0           | 1000            | 0                   | 0       | 0              |         |
| 3            | 172.18.0.20 | 3306 | 0         | ONLINE | 1      | 0           | 1000            | 0                   | 0       | 0              |         |
| 3            | 172.18.0.10 | 3306 | 0         | ONLINE | 1      | 0           | 1000            | 0                   | 0       | 0              |         |
+--------------+-------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
5 rows in set (0.00 sec)
  1. Switch writer node to the pxc MAINTENANCE mode.
MySQL [(none)]> select * from runtime_mysql_servers;
+--------------+-------------+------+-----------+---------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname    | port | gtid_port | status  | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+-------------+------+-----------+---------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 1            | 172.18.0.30 | 3306 | 0         | SHUNNED | 1      | 0           | 1000            | 0                   | 0       | 0              |         |
| 2            | 172.18.0.10 | 3306 | 0         | ONLINE  | 1      | 0           | 1000            | 0                   | 0       | 0              |         |
| 3            | 172.18.0.10 | 3306 | 0         | ONLINE  | 1      | 0           | 1000            | 0                   | 0       | 0              |         |
| 1            | 172.18.0.20 | 3306 | 0         | ONLINE  | 1      | 0           | 1000            | 0                   | 0       | 0              |         |
+--------------+-------------+------+-----------+---------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
4 rows in set (0.00 sec)
  1. Switch writer node back to the pxc DISABLED mode.
MySQL [(none)]> select * from runtime_mysql_servers;
+--------------+-------------+------+-----------+---------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname    | port | gtid_port | status  | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+-------------+------+-----------+---------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 1            | 172.18.0.30 | 3306 | 0         | SHUNNED | 1      | 0           | 1000            | 0                   | 0       | 0              |         |
| 2            | 172.18.0.10 | 3306 | 0         | ONLINE  | 1      | 0           | 1000            | 0                   | 0       | 0              |         |
| 3            | 172.18.0.10 | 3306 | 0         | ONLINE  | 1      | 0           | 1000            | 0                   | 0       | 0              |         |
| 1            | 172.18.0.20 | 3306 | 0         | ONLINE  | 1      | 0           | 1000            | 0                   | 0       | 0              |         |
+--------------+-------------+------+-----------+---------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
4 rows in set (0.00 sec)

@alpes214
Copy link
Contributor Author

@Tusamarco this is a correct PR which fixes the issue #2656. I will close the previous one #2667

@renecannao renecannao merged commit c9c53a1 into v2.0.11 Apr 28, 2020
@renecannao
Copy link
Contributor

Cherry-picked in 2.1.0

@rameshvs02
Copy link

@renecannao I think the issue is still not resolved in 2.0.11

mysql> select hostgroup_id,hostname,port,status from runtime_mysql_servers;
+--------------+----------------+------+--------+
| hostgroup_id | hostname       | port | status |
+--------------+----------------+------+--------+
| 10           | 192.168.100.30 | 3306 | ONLINE |
| 11           | 192.168.100.20 | 3306 | ONLINE |
| 11           | 192.168.100.10 | 3306 | ONLINE |
| 12           | 192.168.100.20 | 3306 | ONLINE |
| 12           | 192.168.100.10 | 3306 | ONLINE |
+--------------+----------------+------+--------+
5 rows in set (0.00 sec)

mysql> select writer_hostgroup,backup_writer_hostgroup,reader_hostgroup,offline_hostgroup,writer_is_also_reader from runtime_mysql_galera_hostgroups;
+------------------+-------------------------+------------------+-------------------+-----------------------+
| writer_hostgroup | backup_writer_hostgroup | reader_hostgroup | offline_hostgroup | writer_is_also_reader |
+------------------+-------------------------+------------------+-------------------+-----------------------+
| 10               | 12                      | 11               | 13                | 2                     |
+------------------+-------------------------+------------------+-------------------+-----------------------+
1 row in set (0.00 sec)

mysql>

mysql> set global pxc_maint_mode = MAINTENANCE;
Query OK, 0 rows affected (10.00 sec)

mysql> set global pxc_maint_mode = DISABLED;
Query OK, 0 rows affected (0.00 sec)

mysql> select @@pxc_maint_mode;
+------------------+
| @@pxc_maint_mode |
+------------------+
| DISABLED         |
+------------------+
1 row in set (0.00 sec)

mysql> select @@wsrep_node_address;
+----------------------+
| @@wsrep_node_address |
+----------------------+
| 192.168.100.30       |
+----------------------+
1 row in set (0.00 sec)

mysql>

mysql> select hostgroup_id,hostname,port,status from runtime_mysql_servers;
+--------------+----------------+------+---------+
| hostgroup_id | hostname       | port | status  |
+--------------+----------------+------+---------+
| 10           | 192.168.100.30 | 3306 | SHUNNED |
| 11           | 192.168.100.10 | 3306 | ONLINE  |
| 12           | 192.168.100.10 | 3306 | ONLINE  |
| 10           | 192.168.100.20 | 3306 | ONLINE  |
+--------------+----------------+------+---------+
4 rows in set (0.00 sec)

mysql>

proxysql version

[vagrant@proxysql ~]$ proxysql --version
ProxySQL version 2.0.11-124-g971c15e, codename Truls
[vagrant@proxysql ~]$

ProxySQL Error log

2020-05-12 10:41:04 [INFO] Galera: max_writers=1 , moving 1 nodes from backup HG 12 to writer HG 10
2020-05-12 10:41:04 MySQL_HostGroups_Manager.cpp:4425:update_galera_set_offline(): [WARNING] Galera: setting host 192.168.100.30:3306 offline because: pxc_maint_mode=YES
2020-05-12 10:41:04 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
HID: 10 , address: 192.168.100.30 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 12 , address: 192.168.100.10 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 12 , address: 192.168.100.20 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: 192.168.100.10 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: 192.168.100.20 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2020-05-12 10:41:04 [INFO] Dumping mysql_servers: ALL
+-----+----------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| hid | hostname       | port | gtid | weight | status | cmp | max_conns | max_lag | ssl | max_lat | comment | mem_pointer     |
+-----+----------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| 10  | 192.168.100.30 | 3306 | 0    | 1000   | 0      | 0   | 1000      | 0       | 0   | 0       |         | 139884047144704 |
| 11  | 192.168.100.20 | 3306 | 0    | 1000   | 0      | 0   | 1000      | 0       | 0   | 0       |         | 139884047144864 |
| 11  | 192.168.100.10 | 3306 | 0    | 1000   | 0      | 0   | 1000      | 0       | 0   | 0       |         | 139884058742720 |
| 12  | 192.168.100.20 | 3306 | 0    | 1000   | 0      | 0   | 1000      | 0       | 0   | 0       |         | 139884033576576 |
| 12  | 192.168.100.10 | 3306 | 0    | 1000   | 0      | 0   | 1000      | 0       | 0   | 0       |         | 139884058742400 |
+-----+----------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
2020-05-12 10:41:04 [INFO] Dumping mysql_servers_incoming
+--------------+----------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname       | port | gtid_port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+----------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 10           | 192.168.100.30 | 3306 | 0         | 1000   | 1      | 0           | 1000            | 0                   | 0       | 0              |         |
| 10           | 192.168.100.20 | 3306 | 0         | 1000   | 0      | 0           | 1000            | 0                   | 0       | 0              |         |
| 12           | 192.168.100.10 | 3306 | 0         | 1000   | 0      | 0           | 1000            | 0                   | 0       | 0              |         |
| 11           | 192.168.100.10 | 3306 | 0         | 1000   | 0      | 0           | 1000            | 0                   | 0       | 0              |         |
+--------------+----------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
2020-05-12 10:41:04 [INFO] Dumping mysql_servers LEFT JOIN mysql_servers_incoming
+-----------------+--------------+----------------+------+
| mem_pointer     | hostgroup_id | hostname       | port |
+-----------------+--------------+----------------+------+
| 139884047144864 | 11           | 192.168.100.20 | 3306 |
| 139884033576576 | 12           | 192.168.100.20 | 3306 |
+-----------------+--------------+----------------+------+
2020-05-12 10:41:04 MySQL_HostGroups_Manager.cpp:1268:commit(): [WARNING] Removed server at address 139884047144864, hostgroup 11, address 192.168.100.20 port 3306. Setting status OFFLINE HARD and immediately dropping all free connections. Used connections will be dropped when trying to use them
2020-05-12 10:41:04 MySQL_HostGroups_Manager.cpp:1268:commit(): [WARNING] Removed server at address 139884033576576, hostgroup 12, address 192.168.100.20 port 3306. Setting status OFFLINE HARD and immediately dropping all free connections. Used connections will be dropped when trying to use them
2020-05-12 10:41:04 [INFO] Dumping mysql_servers JOIN mysql_servers_incoming
+--------------+----------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname       | port | gtid_port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer     | gtid_port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+----------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 10           | 192.168.100.30 | 3306 | 0         | 1000   | 0      | 0           | 1000            | 0                   | 0       | 0              |         | 139884047144704 | 0         | 1000   | 1      | 0           | 1000            | 0                   | 0       | 0              |         |
| 10           | 192.168.100.20 | 3306 | 0         | 1000   | 0      | 0           | 1000            | 0                   | 0       | 0              |         | 0               | 0         | 1000   | 0      | 0           | 1000            | 0                   | 0       | 0              |         |
+--------------+----------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
2020-05-12 10:41:04 [INFO] Changing status for server 10:192.168.100.30:3306 (192.168.100.30:3306) from 0 (0) to 1
2020-05-12 10:41:04 [INFO] Creating new server in HG 10 : 192.168.100.20:3306 , gtid_port=0, weight=1000, status=0
2020-05-12 10:41:04 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
HID: 10 , address: 192.168.100.30 , port: 3306 , gtid_port: 0 , weight: 1000 , status: SHUNNED , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 10 , address: 192.168.100.20 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 12 , address: 192.168.100.10 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 12 , address: 192.168.100.20 , port: 3306 , gtid_port: 0 , weight: 1000 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: 192.168.100.10 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: 192.168.100.20 , port: 3306 , gtid_port: 0 , weight: 1000 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2020-05-12 10:41:04 [INFO] Dumping mysql_servers: ALL
+-----+----------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| hid | hostname       | port | gtid | weight | status | cmp | max_conns | max_lag | ssl | max_lat | comment | mem_pointer     |
+-----+----------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| 10  | 192.168.100.30 | 3306 | 0    | 1000   | 1      | 0   | 1000      | 0       | 0   | 0       |         | 139884047144704 |
| 11  | 192.168.100.20 | 3306 | 0    | 1000   | 3      | 0   | 1000      | 0       | 0   | 0       |         | 139884047144864 |
| 11  | 192.168.100.10 | 3306 | 0    | 1000   | 0      | 0   | 1000      | 0       | 0   | 0       |         | 139884058742720 |
| 12  | 192.168.100.20 | 3306 | 0    | 1000   | 3      | 0   | 1000      | 0       | 0   | 0       |         | 139884033576576 |
| 12  | 192.168.100.10 | 3306 | 0    | 1000   | 0      | 0   | 1000      | 0       | 0   | 0       |         | 139884058742400 |
| 10  | 192.168.100.20 | 3306 | 0    | 1000   | 0      | 0   | 1000      | 0       | 0   | 0       |         | 139884081843840 |
+-----+----------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
2020-05-12 10:41:04 [INFO] MySQL_HostGroups_Manager::commit() locked for 2ms
2020-05-12 10:41:04 [INFO] Dumping current MySQL Servers structures for hostgroup 10
HID: 10 , address: 192.168.100.30 , port: 3306 , gtid_port: 0 , weight: 1000 , status: SHUNNED , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 10 , address: 192.168.100.20 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2020-05-12 10:41:04 [INFO] Dumping mysql_servers: HG 10
+-----+----------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| hid | hostname       | port | gtid | weight | status | cmp | max_conns | max_lag | ssl | max_lat | comment | mem_pointer     |
+-----+----------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| 10  | 192.168.100.20 | 3306 | 0    | 1000   | 0      | 0   | 1000      | 0       | 0   | 0       |         | 139884081843840 |
| 10  | 192.168.100.30 | 3306 | 0    | 1000   | 1      | 0   | 1000      | 0       | 0   | 0       |         | 139884047144704 |
+-----+----------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
2020-05-12 10:41:04 [INFO] Dumping current MySQL Servers structures for hostgroup 12
HID: 12 , address: 192.168.100.10 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 12 , address: 192.168.100.20 , port: 3306 , gtid_port: 0 , weight: 1000 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2020-05-12 10:41:04 [INFO] Dumping mysql_servers: HG 12
+-----+----------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| hid | hostname       | port | gtid | weight | status | cmp | max_conns | max_lag | ssl | max_lat | comment | mem_pointer     |
+-----+----------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| 12  | 192.168.100.10 | 3306 | 0    | 1000   | 0      | 0   | 1000      | 0       | 0   | 0       |         | 139884058742400 |
| 12  | 192.168.100.20 | 3306 | 0    | 1000   | 3      | 0   | 1000      | 0       | 0   | 0       |         | 139884033576576 |
+-----+----------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
2020-05-12 10:41:04 [INFO] Dumping current MySQL Servers structures for hostgroup 11
HID: 11 , address: 192.168.100.10 , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: 192.168.100.20 , port: 3306 , gtid_port: 0 , weight: 1000 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2020-05-12 10:41:04 [INFO] Dumping mysql_servers: HG 11
+-----+----------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| hid | hostname       | port | gtid | weight | status | cmp | max_conns | max_lag | ssl | max_lat | comment | mem_pointer     |
+-----+----------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
| 11  | 192.168.100.10 | 3306 | 0    | 1000   | 0      | 0   | 1000      | 0       | 0   | 0       |         | 139884058742720 |
| 11  | 192.168.100.20 | 3306 | 0    | 1000   | 3      | 0   | 1000      | 0       | 0   | 0       |         | 139884047144864 |
+-----+----------------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-----------------+
2020-05-12 10:41:04 [INFO] Dumping current MySQL Servers structures for hostgroup 13
2020-05-12 10:41:04 [INFO] Dumping mysql_servers: HG 13
+-----+----------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-------------+
| hid | hostname | port | gtid | weight | status | cmp | max_conns | max_lag | ssl | max_lat | comment | mem_pointer |
+-----+----------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-------------+
+-----+----------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------+-------------+
2020-05-12 10:41:09 MySQL_HostGroups_Manager.cpp:4455:update_galera_set_offline(): [WARNING] Galera: skipping setting offline node 192.168.100.30:3306 from hostgroup 10 because won't change the list of ONLINE nodes
2020-05-12 10:41:14 MySQL_HostGroups_Manager.cpp:4455:update_galera_set_offline(): [WARNING] Galera: skipping setting offline node 192.168.100.30:3306 from hostgroup 10 because won't change the list of ONLINE nodes
2020-05-12 10:41:19 MySQL_HostGroups_Manager.cpp:4455:update_galera_set_offline(): [WARNING] Galera: skipping setting offline node 192.168.100.30:3306 from hostgroup 10 because won't change the list of ONLINE nodes
2020-05-12 10:41:24 MySQL_HostGroups_Manager.cpp:4455:update_galera_set_offline(): [WARNING] Galera: skipping setting offline node 192.168.100.30:3306 from hostgroup 10 because won't change the list of ONLINE nodes
2020-05-12 10:41:29 MySQL_HostGroups_Manager.cpp:4455:update_galera_set_offline(): [WARNING] Galera: skipping setting offline node 192.168.100.30:3306 from hostgroup 10 because won't change the list of ONLINE nodes

@renecannao renecannao deleted the v2.0.11-galera-pxc-maint-mode branch June 15, 2020 19:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@alpes214 @renecannao @rameshvs02