-
Notifications
You must be signed in to change notification settings - Fork 729
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Graphite metrics provider #915
Add Graphite metrics provider #915
Conversation
This adds a Graphite metric provider to address issue fluxcd#911. Co-authored-by: Mike Ball <[email protected]> Co-authored-by: Nathan Mische <[email protected]> Signed-off-by: Mike Ball <[email protected]> Signed-off-by: Nathan Mische <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please create a section for Graphite in the metrics docs https://github.com/fluxcd/flagger/blob/main/docs/gitbook/usage/metrics.md
Codecov Report
@@ Coverage Diff @@
## main #915 +/- ##
==========================================
+ Coverage 56.78% 56.89% +0.10%
==========================================
Files 69 70 +1
Lines 5727 5825 +98
==========================================
+ Hits 3252 3314 +62
- Misses 1990 2020 +30
- Partials 485 491 +6
Continue to review full report at Codecov.
|
fa74b96
to
560fe14
Compare
This adds documentation of the Graphite metrics provider in support of addressing issue fluxcd#911. Co-authored-by: Mike Ball <[email protected]> Co-authored-by: Nathan Mische <[email protected]> Signed-off-by: Mike Ball <[email protected]> Signed-off-by: Nathan Mische <[email protected]>
560fe14
to
6a66113
Compare
docs/gitbook/usage/metrics.md
Outdated
target=summarize( | ||
asPercent( | ||
sumSeries( | ||
stats.timers.httpServerRequests.exception.*.method.*.outcome.{CLIENT_ERROR,INFORMATIONAL,REDIRECTION,SUCCESS}.status.*.uri.*.count |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Was this tested? I haven't used Graphite in ages but this query seem wrong to me, also it doesn't filter the canary pods.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@stefanprodan: We have updated the query to include the {{target}}
parameter. One question we do have, is the query intended to consider metrics for both the primary and canary pods, or just the canary pods?
As for the query's correctness, we went ahead and set up a demo Spring Boot app that ships metrics to graphite to test this out. You can find the example here: https://github.com/nmische/spring-boot-graphite. Based on some testing with that application we believe this query correctly calculates the request success rate using the default metrics generated by Spring Boot (with an additional tag app
configured to identify the application).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The builtin checks are for the canary pods only, my assumption is that users don't want a canary analysis to fail if the older version (primary) is running into errors, imagine you have some bug in production, if you look at both versions you'll never manage to deliver a fix.
Assuming that your app uses the Kubernetes Deployment name when it reports to Graphite, then the current query is ok, as the primary app name will be {{target}}-primary
thus its metrics are not included in the query.
3788139
to
c99ac18
Compare
Co-authored-by: Mike Ball <[email protected]> Co-authored-by: Nathan Mische <[email protected]> Signed-off-by: Mike Ball <[email protected]> Signed-off-by: Nathan Mische <[email protected]>
c99ac18
to
8e3ee34
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Thanks Mike & Nathan 🏅
One last thing missing: in here https://github.com/fluxcd/flagger/blob/main/artifacts/flagger/crd.yaml#L1104 there should be |
…te crd Co-authored-by: Mike Ball <[email protected]> Co-authored-by: Nathan Mische <[email protected]> Signed-off-by: Mike Ball <[email protected]> Signed-off-by: Nathan Mische <[email protected]>
This adds a Graphite metric provider, closes #911.
Co-authored-by: Mike Ball [email protected]
Co-authored-by: Nathan Mische [email protected]
Signed-off-by: Mike Ball [email protected]
Signed-off-by: Nathan Mische [email protected]