Coprocessor task terminated due to exceeding the deadline in tispark scenario

[lilinghai]

Bug Report

What version of TiKV are you using?

6.5.1

What operating system and CPU are you using?

Steps to reproduce

9 tikvs for voter role, 3 tikvs for learner role, tispark read from learner roles.
tispark job is select * from a_big_table(3T,single replica) and the learner tikv cpu is about 80% during the spark job
The spark job takes about 30 minutes and failed for the error of Coprocessor task terminated due to exceeding the deadline occasionally.
Maybe we can refine ther retry logic for the coprocessor timeout in the tispark or tikv.

What did you expect?

spark job success

What did happened?

[tonyxuqqi]

@lilinghai Did you try to increase the https://docs.pingcap.com/tidb/dev/tikv-configuration-file#end-point-request-max-handle-duration in TiKV?
After this change, if it's still unable to solve this issue, then it's probably related to client side coprocessor timeout. We're making client-side timeout value configuration in configuration file.
pingcap/tidb#45298
tikv/client-go#865

[glkappe]

pingcap/tidb#45298 tikv/client-go#865
I want to make sure they haven't been picked to the release version yet, right?

[tonyxuqqi]

@glkappe It's merged.