Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

schedule: operator.check will meet datarace #8263

Closed
lhy1024 opened this issue Jun 6, 2024 · 0 comments · Fixed by #8264
Closed

schedule: operator.check will meet datarace #8263

lhy1024 opened this issue Jun 6, 2024 · 0 comments · Fixed by #8264
Labels
affects-5.4 affects-6.1 affects-6.5 affects-7.1 affects-7.5 affects-8.1 severity/major type/bug The issue is confirmed as a bug.

Comments

@lhy1024
Copy link
Contributor

lhy1024 commented Jun 6, 2024 •
edited
Loading

Bug Report

What did you do?

run a server which built with -race

What did you expect to see?

no datarace

What did you see instead?

WARNING: DATA RACE
Read at 0x00c059c28fc8 by goroutine 2253766:
  github.com/tikv/pd/pkg/schedule/operator.(*Operator).Check()
      /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/pkg/schedule/operator/operator.go:382 +0x544
  github.com/tikv/pd/pkg/schedule/operator.(*Controller).pollNeedDispatchRegion()
      /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/pkg/schedule/operator/operator_controller.go:262 +0x144
  github.com/tikv/pd/pkg/schedule/operator.(*Controller).PushOperators()
      /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/pkg/schedule/operator/operator_controller.go:281 +0x2e
  github.com/tikv/pd/pkg/schedule.(*Coordinator).drivePushOperator()
      /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/pkg/schedule/coordinator.go:412 +0x224
  github.com/tikv/pd/pkg/schedule.(*Coordinator).Run.func5()
      /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/pkg/schedule/coordinator.go:491 +0x33

Previous write at 0x00c059c28fc8 by goroutine 37567:
  sync/atomic.CompareAndSwapInt64()
      /usr/local/go/src/runtime/race_amd64.s:316 +0xb
  sync/atomic.CompareAndSwapInt64()
      <autogenerated>:1 +0x1a
  github.com/tikv/pd/pkg/schedule/operator.(*Controller).Dispatch()
      /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/pkg/schedule/operator/operator_controller.go:145 +0x104
  github.com/tikv/pd/server/cluster.(*RaftCluster).HandleRegionHeartbeat()
      /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/server/cluster/cluster_worker.go:66 +0x5c6  github.com/tikv/pd/server.(*GrpcServer).RegionHeartbeat()
      /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/server/grpc_service.go:1296 +0x1c59
  github.com/pingcap/kvproto/pkg/pdpb._PD_RegionHeartbeat_Handler()
      /go/pkg/mod/github.com/pingcap/[email protected]/pkg/pdpb/pdpb.pb.go:9777 +0xb8
  github.com/grpc-ecosystem/go-grpc-prometheus.init.(*ServerMetrics).StreamServerInterceptor.func4()
      /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/server_metrics.go:121 +0x15e
  go.etcd.io/etcd/etcdserver/api/v3rpc.Server.ChainStreamServer.func9.1()
      /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:83 +0xc3
  go.etcd.io/etcd/etcdserver/api/v3rpc.newStreamInterceptor.func1()
      /go/pkg/mod/go.etcd.io/[email protected]/etcdserver/api/v3rpc/interceptor.go:238 +0x6f0
  go.etcd.io/etcd/etcdserver/api/v3rpc.Server.ChainStreamServer.func9()
      /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:86 +0x218
  google.golang.org/grpc.(*Server).processStreamingRPC()
      /go/pkg/mod/google.golang.org/[email protected]/server.go:1687 +0x1f50
  google.golang.org/grpc.(*Server).handleStream()
      /go/pkg/mod/google.golang.org/[email protected]/server.go:1801 +0x19f3
  google.golang.org/grpc.(*Server).serveStreams.func2.1()
      /go/pkg/mod/google.golang.org/[email protected]/server.go:1027 +0x15c

Goroutine 2253766 (running) created at:
  github.com/tikv/pd/pkg/schedule.(*Coordinator).Run()
      /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/pkg/schedule/coordinator.go:491 +0x44a
  github.com/tikv/pd/pkg/schedule.(*Coordinator).RunUntilStop()
      /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/pkg/schedule/coordinator.go:455 +0x48
  github.com/tikv/pd/server/cluster.(*schedulingController).runCoordinator()
      /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/server/cluster/scheduling_controller.go:121 +0xe6
  github.com/tikv/pd/server/cluster.(*schedulingController).startSchedulingJobs.func2()
      /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/server/cluster/scheduling_controller.go:98 +0x33

Goroutine 37567 (running) created at:
  google.golang.org/grpc.(*Server).serveStreams.func2()
      /go/pkg/mod/google.golang.org/[email protected]/server.go:1038 +0x224
  google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders()
      /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:619 +0x3a01
  google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams()
      /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:660 +0x285
  google.golang.org/grpc.(*Server).serveStreams()
      /go/pkg/mod/google.golang.org/[email protected]/server.go:1021 +0x6bb
  google.golang.org/grpc.(*Server).handleRawConn.func1()
      /go/pkg/mod/google.golang.org/[email protected]/server.go:957 +0x86
==================

What version of PD are you using (pd-server -V)?
@lhy1024 lhy1024 added the type/bug The issue is confirmed as a bug. label Jun 6, 2024
@lhy1024 lhy1024 mentioned this issue Jun 6, 2024
Merged
@HuSharp HuSharp mentioned this issue Jun 7, 2024
Merged
@ti-chi-bot ti-chi-bot bot closed this as completed in #8264 Jun 7, 2024
ti-chi-bot bot pushed a commit that referenced this issue Jun 7, 2024
@lhy1024
schedule: fix datarace in operator.check (#8264)
e767c01 
close #8263

Signed-off-by: lhy1024 <[email protected]>
ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this issue Jul 16, 2024
@lhy1024 @ti-chi-bot
This is an automated cherry-pick of tikv#8264
d9684f8 
close tikv#8263

Signed-off-by: ti-chi-bot <[email protected]>
@ti-chi-bot ti-chi-bot mentioned this issue Jul 16, 2024
Merged
ti-chi-bot bot pushed a commit that referenced this issue Jul 19, 2024
@ti-chi-bot @lhy1024
schedule: fix datarace in operator.check (#8264) (#8401)
1bb96d4 
close #8263

Signed-off-by: lhy1024 <[email protected]>

Co-authored-by: lhy1024 <[email protected]>
@ti-chi-bot ti-chi-bot mentioned this issue Jul 31, 2024
Merged
ti-chi-bot bot added a commit that referenced this issue Aug 1, 2024
@ti-chi-bot @lhy1024 @ti-chi-bot
schedule: fix datarace in operator.check (#8264) (#8469)
85258b3 
close #8263

Signed-off-by: lhy1024 <[email protected]>

Co-authored-by: lhy1024 <[email protected]>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this issue Aug 27, 2024
@lhy1024 @ti-chi-bot
This is an automated cherry-pick of tikv#8264
dc5daf2 
close tikv#8263

Signed-off-by: ti-chi-bot <[email protected]>
This was referenced Aug 27, 2024
Merged
Merged
ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this issue Aug 27, 2024
@lhy1024 @ti-chi-bot
This is an automated cherry-pick of tikv#8264
5675fe3 
close tikv#8263

Signed-off-by: ti-chi-bot <[email protected]>
ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this issue Aug 27, 2024
@lhy1024 @ti-chi-bot
This is an automated cherry-pick of tikv#8264
247158f 
close tikv#8263

Signed-off-by: ti-chi-bot <[email protected]>
@ti-chi-bot ti-chi-bot mentioned this issue Aug 27, 2024
Closed
ti-chi-bot bot pushed a commit that referenced this issue Sep 4, 2024
@ti-chi-bot @HuSharp
schedule: fix datarace in operator.check (#8264) (#8579)
1bbec47 
close #8263

Signed-off-by: husharp <[email protected]>

Co-authored-by: husharp <[email protected]>
Co-authored-by: Hu# <[email protected]>
ti-chi-bot bot added a commit that referenced this issue Sep 10, 2024
@ti-chi-bot @HuSharp @ti-chi-bot
schedule: fix datarace in operator.check (#8264) (#8578)
9fb2837 
close #8263

Signed-off-by: husharp <[email protected]>

Co-authored-by: husharp <[email protected]>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affects-5.4 affects-6.1 affects-6.5 affects-7.1 affects-7.5 affects-8.1 severity/major type/bug The issue is confirmed as a bug.
Projects
Questions and Bug Reports
Status: Closed
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

schedule: fix datarace in operator.check lhy1024/pd
2 participants
@jebter @lhy1024