kvprober: implement a prober that sends queries to KV in order to generate signal on the healthiness of KV client & below

[joshimhoff]

https://docs.google.com/document/d/1NqsIgizseyMxUBimpE10m6sSnbebQ25VyJRUwxdmJyM/edit

Is your feature request related to a problem? Please describe.
The problem to solve it provide operators & others better signals of the healthiness of kvclient & below.

Operators & others can alert on these signals. CRL can use them to think about how "big" outages are & to understand where in CRDB stack (is KV broken or not?) & keyspace the impact is.

Describe the solution you'd like
Implement a prober that sends queries to KV in order to generate signal on the healthiness of KV client & below. For a POC that does reads only, see #58124.

Describe alternatives you've considered
More discussion in design doc linked above about alternatives.

By 21.2:

• Implement a basic prober that does reads the whole "rangespace" by sending Gets to the start key of a randomly selected ranges in loop: kvprober: introduce a package for probing the KV layer that does reads only #58124
• Protect CRDB against repeated plan failures: kvprober: avoid repeated planning failures causing CRDB issues #61255
• Probe start key + some suffix to avoid contention as per comments from Tobias here: kvprober: introduce a package for probing the KV layer that does reads only #58124
• Move to timeutil.NewTimer per comments from Tobias here: kvprober: rate limit the planner #61275
• Address meta2 in more idiomatic way as per comments from Tobias here: kvprober: introduce a package for probing the KV layer that does reads only #58124
• Design v1 write prober (prob custom command to send NOP operation thru raft): kvprober: implement "shadow write" probes #67112
• Implement v1 write prober: kvprober: implement "shadow write" probes #67112

Later:

• Log out successful probes but without adding so much noise to logs, e.g. by having a dedicated log file for kvprober
• Design v1 scan prober (scan at historical timestamp)
• Implement v1 scan prober
• Break metrics down by leaseholder
• Trace all probes made by kvprober: tracing: export kvprober traces to a dedicated log file #69427
• Do structured logging via pkg/util/log/eventpb
• Test panic recovery
• Test timeout logic
• Build a probe whole keyspace fast mode: kvprober: allow a manual probe of the entire keyspace #61695
• Build repeatedly probe unavailable ranges once found logic

Epic CC-5241

[joshimhoff]

I opened some follow up issues. I am closing this one as the main work is DONE!