Add absent alerts

[devopsjonas]

This is part of moving away from https://github.com/devopyio/ceph-monitoring-mixin to this community repo :)

This is how generated alerting rules look:

"groups":
- "name": "ceph-mgr-status"
  "rules":
  - "alert": "CephMgrIsAbsent"
    "annotations":
      "description": "Ceph Manager has disappeared from Prometheus target discovery."
      "message": "Storage metrics collector service not available anymore."
      "severity_level": "warning"
      "storage_type": "ceph"
    "expr": |
      absent(up{job="rook-ceph-mgr"} == 1)
    "for": "5m"
    "labels":
      "severity": "warning"
  - "alert": "CephMgrIsMissingReplicas"
    "annotations":
      "description": "Ceph Manager is missing replicas."
      "message": "Storage metrics collector service not available anymore."
      "severity_level": "warning"
      "storage_type": "ceph"
    "expr": |
      sum(up{job="rook-ceph-mgr"}) != 3
    "for": "5m"
    "labels":
      "severity": "warning"

[anmolsachan]

@devopsjonas Thanks for the contribution. I was curious to know on how you tested your patches ?
And it would be great if you could add a screenshot.

[devopsjonas]

@devopsjonas Thanks for the contribution. I was curious to know on how you tested your patches ?
And it would be great if you could add a screenshot.

@anmolsachan Sure we do it's in prod for one of our clients, I'll add a screenshot for Prometheus UI if that works for you?

Should I do it for all of my PRs and future PRs?

[anmolsachan]

Should I do it for all of my PRs and future PRs?

@devopsjonas That would be great

[shtripat]

Are we always expecting 3 MGR?

[devopsjonas]

So the alert checks if x number of replicas are running.

So cephMgrCount is a config option, where users set how many cephMgrs they are running. It is a config option, so users are expect to change it.

So should we change the default? If yes, what should the number be? Or drop alert entirely?

[shtripat]

From ceph team I heard that one of the mons ceph-mgr runs?
@leseb does Rook provide and option to decide the no of manager pods?

[anmolsachan]

If we want this config to be there. Lets have it as optional config. We can comment it out and if during the actual deployment the user can uncomment it. @shtripat @devopsjonas what do you think ?

[shtripat]

Certainly it should not be default configuration I feel.

[devopsjonas]

Maybe, instead we should just do sum(up{%(cephExporterSelector)s}) < %(cephMgrCount)d and set it to 1 by default. So that becomes sum(up{job="ceph-mgr"}) < 1, which means if we are not scraping metrics we would get alerted. This would work for everyone. WDYT?

[devopsjonas]

I've changed the default to be 1 and alert condition to fire than it's less thant that.

[devopsjonas]

@shtripat @anmolsachan please take another look. Thank you!

[umangachapagain]

@devopsjonas Any specific reason why the alerts are in different groups and not in the same alerts group say ceph-mgr-status ?

[devopsjonas]

@devopsjonas Any specific reason why the alerts are in different groups and not in the same alerts group say ceph-mgr-status ?

nice catch 🥇 fixed

[devopsjonas]

@shtripat @anmolsachan please take another look 🙇‍♂️

[shtripat]

LGTM.

[shtripat]

Merging this now. Please add another PR for unit tests.