Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Data race in TestPreparingProgress #6230

Closed
rleungx opened this issue Mar 27, 2023 · 1 comment · Fixed by #6242
Closed

Data race in TestPreparingProgress #6230

rleungx opened this issue Mar 27, 2023 · 1 comment · Fixed by #6242
Assignees
@bufferflies
Labels
affects-7.1 type/bug The issue is confirmed as a bug. type/ci The issue is related to CI.

Comments

@rleungx
Copy link
Member

rleungx commented Mar 27, 2023

Flaky Test

Which jobs are failing

==================
WARNING: DATA RACE
Write at 0x00c00b837458 by goroutine 178389:
  github.com/tikv/pd/pkg/core/storelimit.(*limit).Reset()
      /home/runner/work/pd/pd/pkg/core/storelimit/store_limit.go:133 +0x3f5
  github.com/tikv/pd/pkg/core/storelimit.(*StoreRateLimit).Reset()
      /home/runner/work/pd/pd/pkg/core/storelimit/store_limit.go:107 +0x8a
  github.com/tikv/pd/pkg/schedule.(*OperatorController).getOrCreateStoreLimit()
      /home/runner/work/pd/pd/pkg/schedule/operator_controller.go:870 +0x26c
  github.com/tikv/pd/pkg/schedule.(*OperatorController).exceedStoreLimitLocked()
      /home/runner/work/pd/pd/pkg/schedule/operator_controller.go:847 +0x69b
  github.com/tikv/pd/pkg/schedule.(*OperatorController).PromoteWaitingOperator()
      /home/runner/work/pd/pd/pkg/schedule/operator_controller.go:355 +0x510
  github.com/tikv/pd/pkg/schedule.(*OperatorController).AddWaitingOperator()
      /home/runner/work/pd/pd/pkg/schedule/operator_controller.go:314 +0x10d1
  github.com/tikv/pd/server/cluster.(*coordinator).runScheduler()
      /home/runner/work/pd/pd/server/cluster/coordinator.go:809 +0x56d
  github.com/tikv/pd/server/cluster.(*coordinator).addScheduler.func2()
      /home/runner/work/pd/pd/server/cluster/coordinator.go:639 +0x47

Previous read at 0x00c00b837458 by goroutine 178493:
  github.com/tikv/pd/pkg/core/storelimit.(*limit).Available()
      /home/runner/work/pd/pd/pkg/core/storelimit/store_limit.go:139 +0x148
  github.com/tikv/pd/pkg/core/storelimit.(*StoreRateLimit).Available()
      /home/runner/work/pd/pd/pkg/core/storelimit/store_limit.go:90 +0x8a
  github.com/tikv/pd/pkg/core.(*StoreInfo).IsAvailable()
      /home/runner/work/pd/pd/pkg/core/store.go:143 +0x236
  github.com/tikv/pd/pkg/schedule/filter.(*StoreStateFilter).exceedRemoveLimit()
      /home/runner/work/pd/pd/pkg/schedule/filter/filters.go:420 +0xb6
  github.com/tikv/pd/pkg/schedule/filter.(*StoreStateFilter).exceedRemoveLimit-fm()
      <autogenerated>:1 +0x29
  github.com/tikv/pd/pkg/schedule/filter.(*StoreStateFilter).anyConditionMatch()
      /home/runner/work/pd/pd/pkg/schedule/filter/filters.go:514 +0xae6
  github.com/tikv/pd/pkg/schedule/filter.(*StoreStateFilter).Source()
      /home/runner/work/pd/pd/pkg/schedule/filter/filters.go:530 +0x199
  github.com/tikv/pd/pkg/schedule/filter.SelectSourceStores.func1.1()
      /home/runner/work/pd/pd/pkg/schedule/filter/filters.go:37 +0x10b
  github.com/tikv/pd/pkg/slice.AllOf[...].func1()
      /home/runner/work/pd/pd/pkg/slice/slice.go:35 +0x4b
  github.com/tikv/pd/pkg/slice.AnyOf[...]()
      /home/runner/work/pd/pd/pkg/slice/slice.go:20 +0xc1
  github.com/tikv/pd/pkg/slice.NoneOf[...]()
      /home/runner/work/pd/pd/pkg/slice/slice.go:29 +0xb8
  github.com/tikv/pd/pkg/slice.AllOf[...]()
      /home/runner/work/pd/pd/pkg/slice/slice.go:37 +0xf8
  github.com/tikv/pd/pkg/schedule/filter.SelectSourceStores.func1()
      /home/runner/work/pd/pd/pkg/schedule/filter/filters.go:36 +0x133
  github.com/tikv/pd/pkg/schedule/filter.filterStoresBy()
      /home/runner/work/pd/pd/pkg/schedule/filter/filters.go:122 +0x216
  github.com/tikv/pd/pkg/schedule/filter.SelectSourceStores()
      /home/runner/work/pd/pd/pkg/schedule/filter/filters.go:35 +0xb9
  github.com/tikv/pd/pkg/schedule/schedulers.(*balanceWitnessScheduler).Schedule()
      /home/runner/work/pd/pd/pkg/schedule/schedulers/balance_witness.go:241 +0xa89
  github.com/tikv/pd/server/cluster.(*scheduleController).Schedule()
      /home/runner/work/pd/pd/server/cluster/coordinator.go:899 +0x31d
  github.com/tikv/pd/server/cluster.(*coordinator).runScheduler()
      /home/runner/work/pd/pd/server/cluster/coordinator.go:808 +0x50a
  github.com/tikv/pd/server/cluster.(*coordinator).addScheduler.func2()
      /home/runner/work/pd/pd/server/cluster/coordinator.go:639 +0x47

Goroutine [1783](https://github.com/tikv/pd/actions/runs/4530971783/jobs/7980551492#step:5:1784)89 (running) created at:
  github.com/tikv/pd/server/cluster.(*coordinator).addScheduler()
      /home/runner/work/pd/pd/server/cluster/coordinator.go:639 +0x3b5
  github.com/tikv/pd/server/cluster.(*coordinator).run()
      /home/runner/work/pd/pd/server/cluster/coordinator.go:403 +0x2e7d
  github.com/tikv/pd/server/cluster.(*coordinator).runUntilStop()
      /home/runner/work/pd/pd/server/cluster/coordinator.go:306 +0x75
  github.com/tikv/pd/server/cluster.(*RaftCluster).runCoordinator()
      /home/runner/work/pd/pd/server/cluster/cluster.go:576 +0x11b
  github.com/tikv/pd/server/cluster.(*RaftCluster).Start.func2()
      /home/runner/work/pd/pd/server/cluster/cluster.go:295 +0x39

Goroutine [1784](https://github.com/tikv/pd/actions/runs/4530971783/jobs/7980551492#step:5:1785)93 (running) created at:
  github.com/tikv/pd/server/cluster.(*coordinator).addScheduler()
      /home/runner/work/pd/pd/server/cluster/coordinator.go:639 +0x3b5
  github.com/tikv/pd/server/cluster.(*coordinator).run()
      /home/runner/work/pd/pd/server/cluster/coordinator.go:403 +0x2e7d
  github.com/tikv/pd/server/cluster.(*coordinator).runUntilStop()
      /home/runner/work/pd/pd/server/cluster/coordinator.go:306 +0x75
  github.com/tikv/pd/server/cluster.(*RaftCluster).runCoordinator()
      /home/runner/work/pd/pd/server/cluster/cluster.go:576 +0x11b
  github.com/tikv/pd/server/cluster.(*RaftCluster).Start.func2()
      /home/runner/work/pd/pd/server/cluster/cluster.go:295 +0x39
==================

CI link

https://github.com/tikv/pd/actions/runs/4530971783/jobs/7980551492

Reason for failure (if possible)

Anything else
@rleungx rleungx added the type/ci The issue is related to CI. label Mar 27, 2023
@bufferflies bufferflies mentioned this issue Mar 29, 2023
Merged
@lhy1024 lhy1024 mentioned this issue Apr 3, 2023
Closed
@rleungx
Copy link
Member Author

rleungx commented Apr 20, 2023 

==================
WARNING: DATA RACE
Read at 0x00c00b470448 by goroutine 233125:
  github.com/tikv/pd/pkg/core/storelimit.(*limit).Available()
      /home/runner/work/pd/pd/pkg/core/storelimit/store_limit.go:142 +0x148
  github.com/tikv/pd/pkg/core/storelimit.(*StoreRateLimit).Available()
      /home/runner/work/pd/pd/pkg/core/storelimit/store_limit.go:93 +0x8a
  github.com/tikv/pd/pkg/core.(*StoreInfo).IsAvailable()
      /home/runner/work/pd/pd/pkg/core/store.go:143 +0x248
  github.com/tikv/pd/pkg/schedule/filter.(*StoreStateFilter).exceedAddLimit()
      /home/runner/work/pd/pd/pkg/schedule/filter/filters.go:436 +0xbc
  github.com/tikv/pd/pkg/schedule/filter.(*StoreStateFilter).exceedAddLimit-fm()
      <autogenerated>:1 +0x57
  github.com/tikv/pd/pkg/schedule/filter.(*StoreStateFilter).anyConditionMatch()
      /home/runner/work/pd/pd/pkg/schedule/filter/filters.go:521 +0xae6
  github.com/tikv/pd/pkg/schedule/filter.(*StoreStateFilter).Target()
      /home/runner/work/pd/pd/pkg/schedule/filter/filters.go:562 +0x30a
  github.com/tikv/pd/pkg/schedule/filter.SelectUnavailableTargetStores.func1.1()
      /home/runner/work/pd/pd/pkg/schedule/filter/filters.go:63 +0x12a
  github.com/tikv/pd/pkg/slice.AnyOf[...]()
      /home/runner/work/pd/pd/pkg/slice/slice.go:20 +0xc1
  github.com/tikv/pd/pkg/schedule/filter.SelectUnavailableTargetStores.func1()
      /home/runner/work/pd/pd/pkg/schedule/filter/filters.go:62 +0x1ed
  github.com/tikv/pd/pkg/schedule/filter.filterStoresBy()
      /home/runner/work/pd/pd/pkg/schedule/filter/filters.go:123 +0x216
  github.com/tikv/pd/pkg/schedule/filter.SelectUnavailableTargetStores()
      /home/runner/work/pd/pd/pkg/schedule/filter/filters.go:60 +0xb9
  github.com/tikv/pd/pkg/schedule/schedulers.(*balanceRegionScheduler).Schedule()
      /home/runner/work/pd/pd/pkg/schedule/schedulers/balance_region.go:136 +0x4ef
  github.com/tikv/pd/server/cluster.(*scheduleController).Schedule()
      /home/runner/work/pd/pd/server/cluster/coordinator.go:900 +0x31d
  github.com/tikv/pd/server/cluster.(*coordinator).runScheduler()
      /home/runner/work/pd/pd/server/cluster/coordinator.go:809 +0x50a
  github.com/tikv/pd/server/cluster.(*coordinator).addScheduler.func2()
      /home/runner/work/pd/pd/server/cluster/coordinator.go:640 +0x47

Previous write at 0x00c00b470448 by goroutine 233098:
  github.com/tikv/pd/pkg/core/storelimit.(*limit).Reset()
      /home/runner/work/pd/pd/pkg/core/storelimit/store_limit.go:136 +0x3f5
  github.com/tikv/pd/pkg/core/storelimit.(*StoreRateLimit).Reset()
      /home/runner/work/pd/pd/pkg/core/storelimit/store_limit.go:110 +0x8a
  github.com/tikv/pd/pkg/schedule.(*OperatorController).getOrCreateStoreLimit()
      /home/runner/work/pd/pd/pkg/schedule/operator_controller.go:883 +0x26c
  github.com/tikv/pd/pkg/schedule.(*OperatorController).ack()
      /home/runner/work/pd/pd/pkg/schedule/operator_controller.go:518 +0x269
  github.com/tikv/pd/pkg/schedule.(*OperatorController).removeOperatorLocked()
      /home/runner/work/pd/pd/pkg/schedule/operator_controller.go:557 +0x639
  github.com/tikv/pd/pkg/schedule.(*OperatorController).RemoveOperator()
      /home/runner/work/pd/pd/pkg/schedule/operator_controller.go:531 +0x13a
  github.com/tikv/pd/pkg/schedule.(*OperatorController).Dispatch()
      /home/runner/work/pd/pd/pkg/schedule/operator_controller.go:125 +0x1474
  github.com/tikv/pd/server/cluster.dispatchHeartbeat()
      /home/runner/work/pd/pd/server/cluster/coordinator_test.go:239 +0x197
  github.com/tikv/pd/server/cluster.TestPeerState()
      /home/runner/work/pd/pd/server/cluster/coordinator_test.go:609 +0xa8f
  testing.tRunner()
      /opt/hostedtoolcache/go/1.20.1/x64/src/testing/testing.go:1576 +0x216
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.20.1/x64/src/testing/testing.go:1629 +0x47

Goroutine 233125 (running) created at:
  github.com/tikv/pd/server/cluster.(*coordinator).addScheduler()
      /home/runner/work/pd/pd/server/cluster/coordinator.go:640 +0x3b5
  github.com/tikv/pd/server/cluster.(*coordinator).run()
      /home/runner/work/pd/pd/server/cluster/coordinator.go:404 +0x2e7d
  github.com/tikv/pd/server/cluster.TestPeerState.func1()
      /home/runner/work/pd/pd/server/cluster/coordinator_test.go:572 +0x2e
  github.com/tikv/pd/server/cluster.prepare()
      /home/runner/work/pd/pd/server/cluster/coordinator_test.go:329 +0x211
  github.com/tikv/pd/server/cluster.TestPeerState()
      /home/runner/work/pd/pd/server/cluster/coordinator_test.go:572 +0x7e
  testing.tRunner()
      /opt/hostedtoolcache/go/1.20.1/x64/src/testing/testing.go:1576 +0x216
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.20.1/x64/src/testing/testing.go:1629 +0x47

Goroutine 233098 (running) created at:
  testing.(*T).Run()
      /opt/hostedtoolcache/go/1.20.1/x64/src/testing/testing.go:1629 +0x805
  testing.runTests.func1()
      /opt/hostedtoolcache/go/1.20.1/x64/src/testing/testing.go:2036 +0x8d
  testing.tRunner()
      /opt/hostedtoolcache/go/1.20.1/x64/src/testing/testing.go:1576 +0x216
  testing.runTests()
      /opt/hostedtoolcache/go/1.20.1/x64/src/testing/testing.go:2034 +0x87c
  testing.(*M).Run()
      /opt/hostedtoolcache/go/1.20.1/x64/src/testing/testing.go:1906 +0xb44
  main.main()
      _testmain.go:258 +0x2fc

https://github.com/tikv/pd/actions/runs/4750182521/jobs/8438130764?pr=6347

ti-chi-bot added a commit that referenced this issue Apr 20, 2023
@bufferflies @ti-chi-bot
fix data race by replace clone (#6242)
0e0313a 
close #6230

Signed-off-by: bufferflies <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>
@bufferflies bufferflies self-assigned this Apr 20, 2023
@bufferflies bufferflies added affects-7.1 type/bug The issue is confirmed as a bug. labels Apr 20, 2023
@ti-chi-bot ti-chi-bot mentioned this issue Apr 20, 2023
Merged
ti-chi-bot added a commit to ti-chi-bot/pd that referenced this issue Apr 20, 2023
@bufferflies @ti-chi-bot
fix data race by replace clone (tikv#6242)
2abe074 
close tikv#6230

Signed-off-by: bufferflies <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>
bufferflies added a commit to bufferflies/pd that referenced this issue Apr 21, 2023
@bufferflies @ti-chi-bot
fix data race by replace clone (tikv#6242)
8d8172e 
close tikv#6230

Signed-off-by: bufferflies <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>
bufferflies added a commit to bufferflies/pd that referenced this issue Apr 21, 2023
@bufferflies @ti-chi-bot
fix data race by replace clone (tikv#6242)
9664911 
close tikv#6230

Signed-off-by: bufferflies <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

fix data race by replace clone (tikv#6242)

close tikv#6230

Signed-off-by: bufferflies <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>
bufferflies added a commit to ti-chi-bot/pd that referenced this issue Apr 21, 2023
@bufferflies @ti-chi-bot
fix data race by replace clone (tikv#6242)
57bea4e 
close tikv#6230

Signed-off-by: bufferflies <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

fix data race by replace clone (tikv#6242)

close tikv#6230

Signed-off-by: bufferflies <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>
bufferflies added a commit to ti-chi-bot/pd that referenced this issue Apr 21, 2023
@bufferflies @ti-chi-bot
fix data race by replace clone (tikv#6242)
a1ab3fb 
close tikv#6230

Signed-off-by: bufferflies <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>
ti-chi-bot added a commit that referenced this issue Apr 21, 2023
@ti-chi-bot @bufferflies
fix data race by replace clone (#6242) (#6350)
2695008 
close #6230, ref #6242

Signed-off-by: bufferflies <[email protected]>

Co-authored-by: buffer <[email protected]>
Co-authored-by: Ti Chi Robot <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bufferflies bufferflies

Labels
affects-7.1 type/bug The issue is confirmed as a bug. type/ci The issue is related to CI.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix data race by replace clone bufferflies/pd
2 participants
@bufferflies @rleungx