TestMemberTestSuite failed because of data race

[TonsnakeLin]

Bug Report

What did you do?

run unit test case TestMemberTestSuite with race.

What did you expect to see?

It runs successfully.

What did you see instead?

It appears occasionally.

What version of PD are you using (pd-server -V)?

V6.5.3

[TonsnakeLin]

The race generated when initializeing tsoAllocator failed at (3) bellow. At this moment, am.ResetAllocatorGroup at (1) will read the variable Leadership.keepAliveCancelFunc and allocator.KeepAllocatorLeader will write the variable Leadership.keepAliveCancelFunc concurrently, the race is generated.

func (am *AllocatorManager) campaignAllocatorLeader(
	loopCtx context.Context,
	allocator *LocalTSOAllocator,
	dcLocationInfo *pdpb.GetDCLocationInfoResponse,
	isNextLeader bool,
) {
	defer am.ResetAllocatorGroup(allocator.GetDCLocation())         // (1)
	// Maintain the Local TSO Allocator leader
	go allocator.KeepAllocatorLeader(ctx)                                        // (2)

	if err := allocator.Initialize(int(dcLocationInfo.Suffix)); err != nil {  // (3)
            ...
            return
        }
        ...
}

This issue will leads to panic or leak goroutines after starting pd-server.

How to fix? We should initialize the tsoAllocator when KeepAllocatorLeader goroutine has been started.

[lhy1024]

It has been fixed by #6636, I will pick it to oldder version #6701 #6702 #6703 #6704