scheduler pending commands is too much and Tikv server timeout

[Hanyu2015]

Only one tikv's scheduler pending commands is too much and it's growing continuously, and then data cannot be inserted into database. After I kill this tikv process, the same thing happened on another tikv. When I execute 'select ... from ..', for some table, there is error 'Tikv server timeout', but the other table doesn't.

[Hanyu2015]

There is also one scheduer worker cpu 100% continuously for the tikv .

[eurekaka]

@Hanyu2015 thanks for the report, could you please paste the monitor metrics of the busy tikv server? also, please describe the topology layout of your cluster.

[Hanyu2015]

3 pd, 3 tidb and 18 tikv. which monitor metrics do you need? @eurekaka

[eurekaka]

@Hanyu2015 Could you please export your grafana board of TiKV to a PDF and attach it here so we can check if it is a problem of hot regions.

[eurekaka]

@hicqu please take a look at this problem.

[Hanyu2015]

@eurekaka
tikv.pdf

[eurekaka]

@hicqu FYI

[Hanyu2015]

I find that sql cannot run on the table sessions who causes this issue, and the error log is 'ERROR 9002 (HY000): TIKV server timeout[try again later]'. The tikv in which scheduler pending commands is too much and growing continuously has one scheduler worker cpu 100% continuously, but other 3 scheduler worker cpu is idle. I think in this tikv much command waits the busy cpu instand of running on the idle cpu. Also In tidb.log, there is error log like 'send tikv request error: rpc error: code = DeadlineExceeded desc = context deadline exceeded'.

[Hanyu2015]

I think I find out what leads to this problem, in this tidb log, the conflictTS is an unreasonable value. After I rebuild the table whose tableid is 346, the problem can be resolved. I think this is a bug. @eurekaka

[eurekaka]

@Hanyu2015

• how do you know the conflictTS is unreasonable?
• by rebuild the table, you mean drop the table and recreate it?

@jackysp PTAL

[Hanyu2015]

In the tidb log I upload, conflictTS=18446744073709551615, which is far greater than startTS.
Yes, after drop the table and recreate it.
@eurekaka

[jackysp]

Thanks for your report, @Hanyu2015 !
PTAL @disksing

[shenli]

@Hanyu2015 Could you show me the result of select tidb_version()?

[Hanyu2015]

v2.1.2-1-g8ba8096 @shenli

[shenli]

Could you send us the log of all the tidb-servers? We want to see the logs between 2019-02-28 07:00 ~ 08:00.

[jackysp]

Hi @Hanyu2015 ! There is one thing here to confirm, do you only use SQL to read and write data, and do not bypass TiDB? That is, there is no direct use of TiKV's kv api to read and write.

[Hanyu2015]

only SQL used. @jackysp

[Hanyu2015]

Uploading tidblog.zip…
@shenli

[jackysp]

Many thanks @Hanyu2015 !
Seems the uploading of the log is failed.

[Hanyu2015]

tidblog.zip
@jackysp

[jackysp]

Got it!
Thanks!

[lysu]

@Hanyu2015 hi, current log file seems be generated by grep '2019-02-28 07:00', and it lost some stack log after 'dispatch error:' that without date time printed, could you send use result grep -A90 '2019-02-28 07:00' tidb2.log again~ thanks~

[jebter]

The issue has not been updated for too long, so I will close it. If there are any updates, you can reopen it.