Continuous profiling causes SQL P999 duration jitter

[djshow832]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. Enable continuous profiling on TiDB
2. Create 5k connections to TiDB
3. Send queries of about 1k QPS
4. Watch the P999 duration of queries

2. What did you expect to see? (Required)

No jitter

3. What did you see instead (Required)

Some duration jitters, including compiling duration, get TSO duration, etc.
P999 sometimes up to 100ms or above.
The jitter happens one time every minute, which matches the frequency of continuous profiling.

4. What is your TiDB version? (Required)

v6.5.4

[djshow832]

I can stably reproduce this on both v6.5.4 and v7.4.0. It only occurs when:

• There are thousands of connections (or goroutines), even if they are idle.
• Continuous profiling instead of manual profiling.

[djshow832]

When collecting the goroutines, there are 4 STW(all goroutines stack trace) events that take totally around 300ms:

[djshow832]

This may be because of golang/go#33250, which has been fixed in https://go-review.googlesource.com/c/go/+/387415 by doing less work during STW.
However, TiDB v7.4.0 uses go 1.21.0 and has this PR merged, but this issue still happens.

[djshow832]

With manual profiling, I can see an STW(goroutine profile) of around 1ms at the beginning of goroutineProfileWithLabelsConcurrent, which is expected.
But I didn't see STW(all goroutines stack trace) as the continuous profiling shows. This kind of STW is only called by runtime.Stack([]byte, bool).

[djshow832]

When I fetch the goroutines with ?debug=2, it becomes quite slow and the trace shows STW(all goroutines stack trace). This is because it fetches goroutine stacks instead of calling runtime_goroutineProfileWithLabels.
https://github.com/golang/go/blob/go1.21.0/src/runtime/pprof/pprof.go#L692

[djshow832]

I have tested with the nightly TiDB cluster and the bug disappeared.