server: enable collecting CPU profiles at a lower rate to limit the CPU overhead of the profiling

[knz]

We'd like to dump periodic CPU profiles on every node. Or at least when CPU usage increases with spikes. That is, we'd like to reuse a similar logic at the one we already use to collect heap dumps and goroutine dumps (#75799).

Unfortunately, the pprof default profile rate (100Hz) is causing a noticeable (1-2%) performance dip.
Given that we usually need profiles when CPU is overloaded, the additional cost due to profiling is unwelcome.

So we'd like to explore a way to collect profiles at a lower sampling rate, to lower the overhead.

Sadly, the code in pprof.StartCPUProfile() which we currently use, hardcodes the rate at 100Hz.

We haven't yet found another way to do this short of forking pprof.

Jira issue: CRDB-12842

[knz]

We're going to add this as one more item that can motivate a custom go runtime extension.

[tbg]

related: #75799

[knz]

@felixge do you happen to have ideas of APIs we can use short of forking pprof?

[felixge]

@knz hey, sorry for the late reply, I'm digging my way out of a huge email backlog right now 🙈.

You should be able to call runtime.SetCPUProfileRate(10) for reducing the sampling rate by 10x. Calling pprof.StartCPUProfile() afterwards will print a warning, but the warning is wrong and can be ignored. The requested sample rate should still work.

See golang/go#42502 for more details and upcoming changes to this API