TiFlash cannot read from normal table after insert and being available on arm64 / mac m1 #6659
Labels
Comments
12 tasks
|
I fail to reproduce on my mac m1. Can you always get the error? |
mysql> use test
Database changed
mysql> create table t (a int primary key, b varchar(255), c int, key (b), key (c,b));
Query OK, 0 rows affected (0.16 sec)
mysql> alter table t set tiflash replica 1;
Query OK, 0 rows affected (0.37 sec)
mysql> select sleep(3);
+----------+
| sleep(3) |
+----------+
| 0 |
+----------+
1 row in set (3.00 sec)
mysql> select * from information_schema.tiflash_replica where table_schema = 'test';
+--------------+------------+----------+---------------+-----------------+-----------+----------+
| TABLE_SCHEMA | TABLE_NAME | TABLE_ID | REPLICA_COUNT | LOCATION_LABELS | AVAILABLE | PROGRESS |
+--------------+------------+----------+---------------+-----------------+-----------+----------+
| test | t | 82 | 1 | | 1 | 1 |
+--------------+------------+----------+---------------+-----------------+-----------+----------+
1 row in set (0.01 sec)
mysql> insert into t values (1,"1",-1);
Query OK, 1 row affected (0.01 sec)
mysql> set @t := 1;
Query OK, 0 rows affected (0.00 sec)
mysql> insert into t select a+@t,a+@t,-(a+@t) from t;
Query OK, 1 row affected (0.01 sec)
Records: 1 Duplicates: 0 Warnings: 0
mysql> set @t := 2 * @t;
Query OK, 0 rows affected (0.00 sec)
mysql> insert into t select a+@t,a+@t,-(a+@t) from t;
Query OK, 2 rows affected (0.00 sec)
Records: 2 Duplicates: 0 Warnings: 0
mysql> set @t := 500000;
Query OK, 0 rows affected (0.00 sec)
mysql> insert into t select a+@t,a+@t,-(a+@t) from t;
Query OK, 4 rows affected (0.00 sec)
Records: 4 Duplicates: 0 Warnings: 0
mysql> set @t := 1000000;
Query OK, 0 rows affected (0.00 sec)
mysql> insert into t select a+@t,a+@t,-(a+@t) from t;
Query OK, 8 rows affected (0.00 sec)
Records: 8 Duplicates: 0 Warnings: 0
mysql> select sleep(3);
+----------+
| sleep(3) |
+----------+
| 0 |
+----------+
1 row in set (3.00 sec)
mysql> select /*+ READ_FROM_STORAGE(TIKV[t]) */ count(*) from t;
+----------+
| count(*) |
+----------+
| 16 |
+----------+
1 row in set (0.00 sec)
mysql> select /*+ READ_FROM_STORAGE(TIFLASH[t]) */ count(*) from t;
+----------+
| count(*) |
+----------+
| 16 |
+----------+
1 row in set (0.01 sec) |
No, not every time, but more than 50% |
|
When you get the error, wait for a second, and then rerun |
|
Yes, I can get the error again. Here are some logs snippets: tiflash.log (instance 0): tiflash_tikv.log (instance 0): tikv.log (instance 2, nothing in 0 or 1): |
|
Not sure if the pd.log from the leader is needed, but here it comes as well: |
Seems the available size of tiflash deploy disk is very limited, and the raft log sync is lagged. @mjonss You can try adding a config to workaround this issue: tiflash/etc/config-template.toml Lines 49 to 55 in 1870004 Or deploy tiflash into a disk that has enough available space. |
|
When the raft log sync is lagged, the progress shown in @mjonss Can you wait for a minute after the tiflash query raise region error and check the |
|
So I assume this bug is related to free disk available on the TiFlash nodes. Having enough free space does not trigger this issue. |
PD has |
|
The progress shown in |
It should be less than |
|
Can no longer reproduce, probably issue with disk space. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Bug Report
Please answer these questions before submitting your issue. Thanks!
I cannot do
select /*+ READ_FROM_STORAGE(TIFLASH[t]) */ count(*) from ton my macbook air m1 (on Linux x86_64 it works)1. Minimal reproduce step (Required)
Start a playground cluster:
Run these sql statements:
2. What did you expect to see? (Required)
Answer from TiFlash
3. What did you see instead (Required)
error:
4. What is your TiFlash version? (Required)
From tiup:
From information_schema.cluster_info:
The text was updated successfully, but these errors were encountered: