br, after flashback, pitr restore data to downstram, data inconsistent with upstream

[seiya-annie]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1.install tidb cluster with tiflash, CDC node
2. start pitr task, CDC task, and set tiflash replica
3. prepare 300G data
4. run workload for 1 hour
5. flashback to 1 hour ago
6. restore data to downstream

2. What did you expect to see? (Required)

flashback successfully
data in downstream is consistent with upstream

3. What did you see instead (Required)

MySQL [flashback]> admin check table user_data5 ;
ERROR 8134 (HY000): data inconsistency in table: user_data5, index: k_1, col: update_time, handle: "26883", index-values:"KindMysqlTime 2022-12-02 10:42:52" != record-values:"KindNull <nil>", compare err:<nil>
MySQL [flashback]> admin check table user_data6 ;
Query OK, 0 rows affected (0.84 sec)

MySQL [flashback]> admin check table user_data9 ;
ERROR 8134 (HY000): data inconsistency in table: user_data9, index: k_1, col: update_time, handle: "58386", index-values:"KindMysqlTime 2022-12-02 10:41:55" != record-values:"KindNull <nil>", compare err:<nil>
MySQL [flashback]> admin check table user_data10;
Query OK, 0 rows affected (2.18 sec)

in upstream:

MySQL [flashback]> select count(*) from user_data5 ;
+----------+
| count(*) |
+----------+
|   350000 |
+----------+
1 row in set (0.43 sec)

MySQL [flashback]> select count(*) from user_data9 ;
+----------+
| count(*) |
+----------+
|   350000 |
+----------+
1 row in set (0.37 sec)

in downstream:

MySQL [flashback]> select count(*) from user_data5 ;
+----------+
| count(*) |
+----------+
|   367630 |
+----------+
1 row in set (0.16 sec)
MySQL [flashback]> select count(*) from user_data9 ;
+----------+
| count(*) |
+----------+
|   361240 |
+----------+
1 row in set (0.16 sec)

4. What is your TiDB version? (Required)

[root@localhost zak]# ./br -V
Release Version: v6.5.0-alpha-201-g8f61a6fea
Git Commit Hash: 8f61a6f
Git Branch: fix-rewrite-shortvalue
Go Version: go1.19
UTC Build Time: 2022-12-02 10:26:54
Race Enabled: false

[seiya-annie]

[root@localhost zak]# nohup ./br restore point --pd=172.16.6.35:3679 --storage="s3://tmp/br-lt-log-backup-1201?access-key=minioadmin&secret-access-key=minioadmin&endpoint=http%3a%2f%2fminio.pingcap.net%3a9000&force-path-style=true" --restored-ts="2022-12-02 19:05:00+0800" --concurrency=64 --pitr-concurrency=128 --full-backup-storage="s3://tmp/backup-46-snapshot-20221201200000?access-key=minioadmin&secret-access-key=minioadmin&endpoint=http%3a%2f%2fminio.pingcap.net%3a9000&force-path-style=true" >> restore-46.log 2>&1 &

[seiya-annie]

br.log.2022-12-02T19.44.50+0800.gz