leases: Add metrics to etcd leases #9778
Conversation
|
Given #9764, I'm not sure whether these should live in the |
|
We will decide whether to keep |
Codecov Report
@@ Coverage Diff @@
## master #9778 +/- ##
=========================================
Coverage ? 69.54%
=========================================
Files ? 376
Lines ? 35215
Branches ? 0
=========================================
Hits ? 24489
Misses ? 8962
Partials ? 1764
Continue to review full report at Codecov.
|
|
@gyuho I am not a fan of many metrics. I hope normal users can only need to pay attention to a small set of well thought metrics. Other metrics are for debugging. |
|
@xiang90 Agree. I will go through all metrics, and select a few important ones (e.g. DB size in use). |
lease/metrics.go
Outdated
| Namespace: "etcd_debugging", | ||
| Subsystem: "lease", | ||
| Name: "renewed_total", | ||
| Help: "The total number of renewed leases.", |
There was a problem hiding this comment.
lease renew is not a cluster wide operation, it only goes through the leader node at least for now. needs to add some clarification here.
There was a problem hiding this comment.
I'll update this.
|
lgtm |
lease/metrics.go
Outdated
| @@ -0,0 +1,59 @@ | |||
| // Copyright 2015 The etcd Authors | |||
There was a problem hiding this comment.
I'll update this.
lease/metrics.go
Outdated
| Name: "total_ttl", | ||
| Help: "Bucketed histogram of lease TTLs.", | ||
| // 1 second -> 1 month | ||
| Buckets: prometheus.ExponentialBuckets(1, 2, 24), |
There was a problem hiding this comment.
pow(2,23) * 1 == 8388608 seconds == 97 days?
Do we really need this high upper bound?
There was a problem hiding this comment.
Leases can be up to 9E9 seconds, so setting this fairly high seemed reasonable, but I'm happy to decrease it if you'd prefer!
There was a problem hiding this comment.
Then, can you update the comment instead? 1 second -> 3 months?
idiamond-stripe
force-pushed
the
idiamond-add-leases-metric
branch
from
6e06d19
to
d9ab580
Compare
lease/metrics.go
Outdated
| prometheus.HistogramOpts{ | ||
| Namespace: "etcd_debugging", | ||
| Subsystem: "lease", | ||
| Name: "total_ttl", |
There was a problem hiding this comment.
Also rename to ttl_total to be consistent with other metrics.
This patch adds four metrics to the `leases` package for easier debugging.
idiamond-stripe
force-pushed
the
idiamond-add-leases-metric
branch
from
d9ab580
to
0369298
Compare
This patch adds four metrics (leases granted, leases revoked, leases renewed, and the average TTL of leases) to the
leasespackage for easier debugging and visibility into outstanding leases.There's not currently a way to monitor the number of leases that are granted or revoked within etcd. It's currently possible to see the number of leases expired via
server_lease_expired_total, but not the number initially granted nor those being renewed.This makes it tough to determine the current number of leases held, and determine why leases are not being cleaned up. This patch makes the behavior described in, for example, #9395 easier to debug and understand as it makes it clear that leases are never being revoked, rather than renewed forever.
Please read https://github.com/coreos/etcd/blob/master/CONTRIBUTING.md#contribution-flow.