br: add metrics for volume backup (#5534)

[ti-chi-bot]

This is an automated cherry-pick of #5534

What problem does this PR solve?

Add metrics for br federation manager

Closes #5533

What is changed and how does it work?

Add metrics about volume backup status, volume backup total time, volume backup total size, volume backup cleanup status and volume backup cleanup total time.

For volume backup cleanup, there is no status to indicate cleanup failed. So I also add a CleanFailed status to backup and volume backup. If backup cleanup is failed, the backup CR will become CleanFailed status. If one of backup cleanup in data plane is failed, the volume backup CR will become CleanFailed status.

Code changes

• Has Go code change
• Has CI related scripts change

Tests

• Unit test
• E2E test
• Manual test
• No code

Side effects

• Breaking backward compatibility
• Other side effects:

Related changes

• Need to cherry-pick to the release branch
• Need to update the documentation

Release Notes

Please refer to Release Notes Language Style Guide before writing the release note.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: BornChanger

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [BornChanger]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2024-02-07 12:25:55.891407563 +0000 UTC m=+359681.458177445: ☑️ agreed by BornChanger.

[BornChanger]

/retest

[codecov-commenter]

Codecov Report

Attention: 32 lines in your changes are missing coverage. Please review.

Comparison is base (456a027) 61.64% compared to head (49b3ab5) 67.92%.

Additional details and impacted files

@@               Coverage Diff               @@
##           release-1.5    #5545      +/-   ##
===============================================
+ Coverage        61.64%   67.92%   +6.28%     
===============================================
  Files              228      242      +14     
  Lines            29151    33149    +3998     
===============================================
+ Hits             17969    22518    +4549     
+ Misses            9420     8825     -595     
- Partials          1762     1806      +44     

Flag	Coverage Δ
e2e	47.81% <38.09%> (?)
unittest	61.60% <48.21%> (-0.04%)	⬇️