Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Killing TiFlash causes TiDB down #28096

Closed
JaySon-Huang opened this issue Sep 16, 2021 · 3 comments · Fixed by #28110
Closed

Killing TiFlash causes TiDB down #28096

JaySon-Huang opened this issue Sep 16, 2021 · 3 comments · Fixed by #28110
Assignees
Labels
affects-5.2 This bug affects 5.2.x versions. severity/major sig/transaction SIG:Transaction type/bug The issue is confirmed as a bug.

Comments

@JaySon-Huang
Copy link
Contributor

JaySon-Huang commented Sep 16, 2021

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

  • Deploy a cluster with 1 tidb, 3 tikv, 2 tiflash, 1 pd.
  • Run some workload ("schrodinger/bank2")
  • Send kill -9 to a TiFlash instance
  • The TiDB instance get restarted

2. What did you expect to see? (Required)

Killing TiFlash instance won't make TiDB get down

3. What did you see instead (Required)

Killing TiFlash instance DO make TiDB get down. The full logs:
tidb-log.tar.gz

>  cat log/tidb_stderr.log
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x2496f87]

goroutine 1342869 [running]:
github.com/pingcap/tidb/store/copr.balanceBatchCopTask.func1(0xc00105a760, 0xc00b60c6a0, 0x2, 0x2, 0xc00105a6f8, 0xc000672ea0, 0xc048e1b73280e987, 0x2daf1881f3, 0x5fadfa0, 0xc001136068, ...)
	/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/src/github.com/pingcap/tidb/store/copr/batch_coprocessor.go:162 +0x3a7
created by github.com/pingcap/tidb/store/copr.balanceBatchCopTask
	/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/src/github.com/pingcap/tidb/store/copr/batch_coprocessor.go:140 +0x16a5
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x2496f87]

goroutine 36705 [running]:
github.com/pingcap/tidb/store/copr.balanceBatchCopTask.func1(0xc001dea260, 0xc002268d20, 0x2, 0x2, 0xc001dea258, 0xc006847a70, 0xc048e1bc3756cb4a, 0x11bf7b5d5, 0x5fadfa0, 0xc00059f4b0, ...)
	/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/src/github.com/pingcap/tidb/store/copr/batch_coprocessor.go:162 +0x3a7
created by github.com/pingcap/tidb/store/copr.balanceBatchCopTask
	/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/src/github.com/pingcap/tidb/store/copr/batch_coprocessor.go:140 +0x16a5

4. What is your TiDB version? (Required)

mysql> select tidb_version();
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| tidb_version()                                                                                                                                                                                                                                                                                                            |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Release Version: v5.3.0-alpha-116-ge262e5984
Edition: Community
Git Commit Hash: e262e59842fbd0eb23f1da71448eaaf97581c4a8
Git Branch: master
UTC Build Time: 2021-09-15 07:55:38
GoVersion: go1.16.4
Race Enabled: false
TiKV Min Version: v3.0.0-60965b006877ca7234adaced7890d7b029ed1306
Check Table Before Drop: false |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

mysql> select type,version,git_hash,start_time,uptime from information_schema.cluster_info;
+---------+---------------------------------+------------------------------------------+---------------------------+------------------+
| type    | version                         | git_hash                                 | start_time                | uptime           |
+---------+---------------------------------+------------------------------------------+---------------------------+------------------+
| tidb    | 5.3.0-alpha                     | e262e59842fbd0eb23f1da71448eaaf97581c4a8 | 2021-09-16T17:30:00+08:00 | 7m5.451833153s   |
| tidb    | 5.3.0-alpha                     | e262e59842fbd0eb23f1da71448eaaf97581c4a8 | 2021-09-16T17:33:52+08:00 | 3m13.451838005s  |
| tidb    | 5.3.0-alpha                     | e262e59842fbd0eb23f1da71448eaaf97581c4a8 | 2021-09-16T17:33:32+08:00 | 3m33.451839836s  |
| pd      | 5.3.0-alpha                     | ef40841fee7f12c89c7562ae149013fb81a3e42b | 2021-09-16T16:46:25+08:00 | 50m40.451841512s |
| tiflash | 5.3.0-alpha-73-ge4b16122d-dirty | e4b16122d22b7664ca690b67a3d38f7e91a17369 | 2021-09-16T17:33:09+08:00 | 3m56.45184323s   |
| tiflash | 5.3.0-alpha-73-ge4b16122d-dirty | e4b16122d22b7664ca690b67a3d38f7e91a17369 | 2021-09-16T17:08:47+08:00 | 28m18.451846117s |
| tikv    | 5.3.0-alpha                     | 6be3893f7f787b04bf34d99d1369092404ab5cfc | 2021-09-16T16:46:28+08:00 | 50m37.45184788s  |
| tikv    | 5.3.0-alpha                     | 6be3893f7f787b04bf34d99d1369092404ab5cfc | 2021-09-16T16:46:28+08:00 | 50m37.451849806s |
| tikv    | 5.3.0-alpha                     | 6be3893f7f787b04bf34d99d1369092404ab5cfc | 2021-09-16T16:46:28+08:00 | 50m37.451852317s |
+---------+---------------------------------+------------------------------------------+---------------------------+------------------+
@JaySon-Huang JaySon-Huang added the type/bug The issue is confirmed as a bug. label Sep 16, 2021
@JaySon-Huang JaySon-Huang changed the title Killing TiFlash cause TiDB down Killing TiFlash causes TiDB down Sep 16, 2021
@JaySon-Huang
Copy link
Contributor Author

JaySon-Huang commented Sep 16, 2021

I've tried to downgrade the TiDB server.
v5.2.0 has the same problem. But v5.1.1 does not.

@JaySon-Huang
Copy link
Contributor Author

/assign @windtalker

@github-actions
Copy link

Please check whether the issue should be labeled with 'affects-x.y' or 'backport-x.y.z',
and then remove 'needs-more-info' label.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects-5.2 This bug affects 5.2.x versions. severity/major sig/transaction SIG:Transaction type/bug The issue is confirmed as a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants