metrics: Add check for unhealthy stores

[dveeden]

What problem does this PR solve?

When the filesystem of a TiKV node of a v5.3.0 cluster becomes read-only (e.g. due to filesystem errors, or simulated with fsfreeze(8) it seems to store_disconneced_count for 10m and then to store_unhealth_count for another 20m before eventually ending up in store_down_count.

Issue Number: Close #5611

See also pingcap/monitoring#172

What is changed and how does it work?

It looks like there isn't a check for store_unhealth_count, so that's what this PR tries to add.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Code changes

• Has configuration change
• Has HTTP API interfaces changed (Don't forget to add the declarative for the new API)
• Has persistent data change

Side effects

• Possible performance regression
• Increased code complexity
• Breaking backward compatibility

Related changes

• PR to update pingcap/docs/pingcap/docs-cn:
• PR to update pingcap/tiup:
• Need to cherry-pick to the release branch

Release note

None

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• lhy1024
• nolouch

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[codecov]

Codecov Report

Base: 75.76% // Head: 75.68% // Decreases project coverage by -0.08% ⚠️

Coverage data is based on head (90f2af4) compared to base (2b51932).
Patch has no changes to coverable lines.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5612      +/-   ##
==========================================
- Coverage   75.76%   75.68%   -0.09%     
==========================================
  Files         326      326              
  Lines       32386    32386              
==========================================
- Hits        24538    24510      -28     
- Misses       5746     5762      +16     
- Partials     2102     2114      +12     

Flag	Coverage Δ
unittests	75.68% <ø> (-0.09%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
server/api/diagnostic.go	76.92% <0.00%> (-23.08%)	⬇️
server/id/id.go	86.44% <0.00%> (-10.17%)	⬇️
pkg/tempurl/tempurl.go	60.00% <0.00%> (-10.00%)	⬇️
server/storage/kv/etcd_kv.go	79.71% <0.00%> (-5.80%)	⬇️
server/region_syncer/server.go	81.86% <0.00%> (-4.40%)	⬇️
server/cluster/diagnostic_manager.go	72.04% <0.00%> (-4.31%)	⬇️
server/member/member.go	62.10% <0.00%> (-3.16%)	⬇️
pkg/dashboard/adapter/manager.go	79.31% <0.00%> (-2.30%)	⬇️
server/cluster/coordinator.go	72.48% <0.00%> (-1.17%)	⬇️
server/storage/hot_region_storage.go	78.14% <0.00%> (-1.10%)	⬇️
... and 18 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[nolouch]

/merge

[ti-chi-bot]

@nolouch: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: 290fa54

[ti-chi-bot]

@dveeden: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.