changefeed task state is warning and lag is more and more after inject network partition between one of tikv and pdleader

[Lily2025]

What did you do?

1、create changefeed
2、run sysbench
sysbench --db-driver=mysql --time=0 --mysql-db=sysbench_64_7000w --mysql-host=tc-tidb.endless-ha-test-ticdc-tps-2220745-1-732 --mysql-port=4000 --threads=64 --tables=64 --table-size=70000000 --mysql-ignore-errors="2013,1213,1105,1205,8022,8028,9004,9007,1062" --mysql-user=root --report-interval=10 oltp_read_write run

3、inject network partition between one of tikv and pdleader last for 10mins
inject fault：2023-09-04 21:04:50
recover fault：2023-09-04 21:14:50

What did you expect to see?

changefeed lag can recover to be less than 3s after fault recover

What did you see instead?

1、changefeed task state is warning

2、lag is more and more after fault recover

Versions of the cluster

githash：6eaf5f0d4083dcc8c5bde8e6dd1ddb4f0a472ace

current status of DM cluster (execute query-status <task-name> in dmctl)

No response

[Lily2025]

/remove-area dm
/area ticdc

[fubinzh]

/severity major

[Lily2025]

/assign nongfushanquan

[nongfushanquan]

fixed by #9742

[nongfushanquan]

/close

[ti-chi-bot]

@nongfushanquan: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.