Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Data race in election #5434

Closed
rleungx opened this issue Aug 16, 2022 · 0 comments · Fixed by #5573
Closed

Data race in election #5434

rleungx opened this issue Aug 16, 2022 · 0 comments · Fixed by #5573
Labels
type/ci The issue is related to CI.

Comments

@rleungx
Copy link
Member

rleungx commented Aug 16, 2022

Flaky Test

Which jobs are failing

TestMemberTestSuite

CI link

local environment

Reason for failure (if possible)

==================
WARNING: DATA RACE
Read at 0x00c002244d78 by goroutine 358498:
  github.com/tikv/pd/server/election.(*Leadership).Reset()
      /data2/ryan/pd/server/election/leadership.go:233 +0x8d
  github.com/tikv/pd/server/member.(*Member).ResetLeader()
      /data2/ryan/pd/server/member/member.go:199 +0x44
  github.com/tikv/pd/server.(*Server).campaignLeader.func3()
      /data2/ryan/pd/server/server.go:1473 +0x64
  sync.(*Once).doSlow()
      /data2/ryan/go1.19/go/src/sync/once.go:74 +0x101
  sync.(*Once).Do()
      /data2/ryan/go1.19/go/src/sync/once.go:65 +0x46
  github.com/tikv/pd/server.(*Server).campaignLeader.func8()
      /data2/ryan/pd/server/server.go:1469 +0x47
  runtime.deferreturn()
      /data2/ryan/go1.19/go/src/runtime/panic.go:476 +0x32
  github.com/tikv/pd/server.(*Server).leaderLoop()
      /data2/ryan/pd/server/server.go:1388 +0x9da
  github.com/tikv/pd/server.(*Server).startServerLoop.func1()
      /data2/ryan/pd/server/server.go:545 +0x39

Previous write at 0x00c002244d78 by goroutine 359003:
  github.com/tikv/pd/server/election.(*Leadership).Keep()
      /data2/ryan/pd/server/election/leadership.go:140 +0xb5
  github.com/tikv/pd/server/member.(*Member).KeepLeader()
      /data2/ryan/pd/server/member/member.go:154 +0x4f
  github.com/tikv/pd/server.(*Server).campaignLeader.func5()
      /data2/ryan/pd/server/server.go:1418 +0x58

Goroutine 358498 (running) created at:
  github.com/tikv/pd/server.(*Server).startServerLoop()
      /data2/ryan/pd/server/server.go:545 +0x184
  github.com/tikv/pd/server.(*Server).Run()
      /data2/ryan/pd/server/server.go:517 +0x189
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:458 +0x3e6
  fmt.Fscanf()
      /data2/ryan/go1.19/go/src/fmt/scan.go:143 +0xe9
  fmt.Sscanf()
      /data2/ryan/go1.19/go/src/fmt/scan.go:114 +0x164
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:643 +0x26
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:458 +0x3e6
  fmt.Fscanf()
      /data2/ryan/go1.19/go/src/fmt/scan.go:143 +0xe9
  fmt.Sscanf()
      /data2/ryan/go1.19/go/src/fmt/scan.go:114 +0x164
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:643 +0x26
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:458 +0x3e6
  github.com/syndtr/goleveldb/leveldb.(*DB).checkAndCleanFiles()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_util.go:52 +0x1fe
  github.com/syndtr/goleveldb/leveldb.openDB()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:130 +0x8b5
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:458 +0x3e6
  fmt.Fscanf()
      /data2/ryan/go1.19/go/src/fmt/scan.go:143 +0xe9
  fmt.Sscanf()
      /data2/ryan/go1.19/go/src/fmt/scan.go:114 +0x224
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:657 +0x16e
  fmt.(*ss).doScanf()
      /data2/ryan/go1.19/go/src/fmt/scan.go:1230 +0x3fb
  fmt.Fscanf()
      /data2/ryan/go1.19/go/src/fmt/scan.go:143 +0xe9
  fmt.Sscanf()
      /data2/ryan/go1.19/go/src/fmt/scan.go:114 +0x164
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:643 +0x26
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:458 +0x3e6
  fmt.Fscanf()
      /data2/ryan/go1.19/go/src/fmt/scan.go:143 +0xe9
  fmt.Sscanf()
      /data2/ryan/go1.19/go/src/fmt/scan.go:114 +0x164
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:643 +0x26
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:458 +0x3e6
  fmt.Fscanf()
      /data2/ryan/go1.19/go/src/fmt/scan.go:143 +0xe9
  fmt.Sscanf()
      /data2/ryan/go1.19/go/src/fmt/scan.go:114 +0x164
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:643 +0x26
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:458 +0x3e6
  github.com/syndtr/goleveldb/leveldb.(*DB).recoverJournal()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:476 +0xb2
  github.com/syndtr/goleveldb/leveldb.openDB()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:125 +0x899
  github.com/syndtr/goleveldb/leveldb.Open()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:197 +0x35c
  fmt.Fscanf()
      /data2/ryan/go1.19/go/src/fmt/scan.go:143 +0xe9
  fmt.Sscanf()
      /data2/ryan/go1.19/go/src/fmt/scan.go:114 +0x164
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:643 +0x26
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:458 +0x3e6
  github.com/syndtr/goleveldb/leveldb.(*session).recover.func1()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/session.go:134 +0xd0
  runtime.deferreturn()
      /data2/ryan/go1.19/go/src/runtime/panic.go:476 +0x32
  github.com/syndtr/goleveldb/leveldb.Open()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:183 +0x104
  github.com/syndtr/goleveldb/leveldb.OpenFile()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:219 +0x87
  github.com/tikv/pd/server/storage/kv.NewLevelDBKV()
      /data2/ryan/pd/server/storage/kv/levedb_kv.go:33 +0x5b
  github.com/tikv/pd/server/storage.NewHotRegionsStorage()
      /data2/ryan/pd/server/storage/hot_region_storage.go:143 +0x8f
  github.com/tikv/pd/server.(*Server).startServer()
      /data2/ryan/pd/server/server.go:419 +0x1d06
  fmt.Sscanf()
      /data2/ryan/go1.19/go/src/fmt/scan.go:114 +0x164
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:643 +0x26
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:458 +0x3e6
  fmt.Fscanf()
      /data2/ryan/go1.19/go/src/fmt/scan.go:143 +0xe9
  fmt.Sscanf()
      /data2/ryan/go1.19/go/src/fmt/scan.go:114 +0x164
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:643 +0x26
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:458 +0x3e6
  fmt.Fscanf()
      /data2/ryan/go1.19/go/src/fmt/scan.go:143 +0xe9
  fmt.Sscanf()
      /data2/ryan/go1.19/go/src/fmt/scan.go:114 +0x164
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:643 +0x26
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:458 +0x3e6
  github.com/syndtr/goleveldb/leveldb.(*DB).checkAndCleanFiles()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_util.go:52 +0x1fe
  github.com/syndtr/goleveldb/leveldb.openDB()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:130 +0x8b5
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:458 +0x3e6
  fmt.Fscanf()
      /data2/ryan/go1.19/go/src/fmt/scan.go:143 +0xe9
  fmt.Sscanf()
      /data2/ryan/go1.19/go/src/fmt/scan.go:114 +0x224
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:657 +0x16e
  fmt.(*ss).doScanf()
      /data2/ryan/go1.19/go/src/fmt/scan.go:1230 +0x3fb
  fmt.Fscanf()
      /data2/ryan/go1.19/go/src/fmt/scan.go:143 +0xe9
  fmt.Sscanf()
      /data2/ryan/go1.19/go/src/fmt/scan.go:114 +0x164
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:643 +0x26
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:458 +0x3e6
  fmt.Fscanf()
      /data2/ryan/go1.19/go/src/fmt/scan.go:143 +0xe9
  fmt.Sscanf()
      /data2/ryan/go1.19/go/src/fmt/scan.go:114 +0x164
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:643 +0x26
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:458 +0x3e6
  fmt.Fscanf()
      /data2/ryan/go1.19/go/src/fmt/scan.go:143 +0xe9
  fmt.Sscanf()
      /data2/ryan/go1.19/go/src/fmt/scan.go:114 +0x164
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:643 +0x26
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:458 +0x3e6
  github.com/syndtr/goleveldb/leveldb.(*DB).recoverJournal()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:476 +0xb2
  github.com/syndtr/goleveldb/leveldb.openDB()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:125 +0x899
  github.com/syndtr/goleveldb/leveldb.Open()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:197 +0x35c
  fmt.Fscanf()
      /data2/ryan/go1.19/go/src/fmt/scan.go:143 +0xe9
  fmt.Sscanf()
      /data2/ryan/go1.19/go/src/fmt/scan.go:114 +0x164
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:643 +0x26
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:458 +0x3e6
  github.com/syndtr/goleveldb/leveldb.(*session).recover.func1()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/session.go:134 +0xd0
  runtime.deferreturn()
      /data2/ryan/go1.19/go/src/runtime/panic.go:476 +0x32
  github.com/syndtr/goleveldb/leveldb.Open()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:183 +0x104
  github.com/syndtr/goleveldb/leveldb.OpenFile()
      /data2/ryan/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:219 +0x87
  github.com/tikv/pd/server/storage/kv.NewLevelDBKV()
      /data2/ryan/pd/server/storage/kv/levedb_kv.go:33 +0x5b
  github.com/tikv/pd/server/storage.newLevelDBBackend()
      /data2/ryan/pd/server/storage/leveldb_backend.go:62 +0x6f
  github.com/tikv/pd/server/storage.NewStorageWithLevelDBBackend()
      /data2/ryan/pd/server/storage/storage.go:62 +0x1204
  github.com/tikv/pd/server.(*Server).startServer()
      /data2/ryan/pd/server/server.go:408 +0x110a
  github.com/tikv/pd/server.(*Server).Run()
      /data2/ryan/pd/server/server.go:513 +0x145
  github.com/tikv/pd/server/api.mustNewCluster.func1()
      /data2/ryan/pd/server/api/server_test.go:97 +0x184
  github.com/tikv/pd/server/api.mustNewCluster.func3()
      /data2/ryan/pd/server/api/server_test.go:100 +0x47

Goroutine 359003 (finished) created at:
  github.com/tikv/pd/server.(*Server).campaignLeader()
      /data2/ryan/pd/server/server.go:1418 +0x877
  github.com/tikv/pd/server.(*Server).leaderLoop()
      /data2/ryan/pd/server/server.go:1388 +0x9da
  github.com/tikv/pd/server.(*Server).startServerLoop.func1()
      /data2/ryan/pd/server/server.go:545 +0x39
==================

Anything else
@rleungx rleungx added the type/ci The issue is related to CI. label Aug 16, 2022
@rleungx rleungx mentioned this issue Oct 9, 2022
Merged
ti-chi-bot added a commit that referenced this issue Oct 9, 2022
@rleungx @ti-chi-bot
*: fix election race (#5573)
d1902f9 
close #5434

Signed-off-by: Ryan Leung <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>
lhy1024 pushed a commit to ti-chi-bot/pd that referenced this issue Jun 28, 2023
@rleungx @ti-chi-bot @lhy1024
*: fix election race (tikv#5573)
0275cba 
close tikv#5434

Signed-off-by: Ryan Leung <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>
Signed-off-by: lhy1024 <[email protected]>
lhy1024 pushed a commit to ti-chi-bot/pd that referenced this issue Jun 28, 2023
@rleungx @ti-chi-bot @lhy1024
*: fix election race (tikv#5573)
f415707 
close tikv#5434

Signed-off-by: Ryan Leung <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>
Signed-off-by: lhy1024 <[email protected]>
lhy1024 pushed a commit to ti-chi-bot/pd that referenced this issue Jun 28, 2023
@rleungx @ti-chi-bot @lhy1024
*: fix election race (tikv#5573)
dbe5909 
close tikv#5434

Signed-off-by: Ryan Leung <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>
Signed-off-by: lhy1024 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/ci The issue is related to CI.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

*: fix election race rleungx/pd
1 participant
@rleungx