Add flag to set custom buckets for prometheus histogram #7171

serge-r · 2021-05-27T07:58:42Z

This PR adding a support for custom sets of prometheus histogram bucket.

What this PR does / why we need it:

It can also resolve issue #3898. Sometimes we needs to compare ingress requests\responses metrics with other applications metrics and need to set up same buckets sets.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Which issue/s this PR fixes

fixes ##3898

How Has This Been Tested?

I have add function to test flags. Also I have tests into default kind environment (make dev-env) and just change nginx-ingress-controller start flags by adding
- --metrics-buckets=[0.05, 0.1, 0.2, 0.3, 0.5, 1.0, 5, 10]

Also I have add test ingress object into dev-env make queries and get this result:

nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-69fbf79577-d5q2t",host="testapp.local",ingress="test-ingress",method="GET",namespace="test",path="/",service="",status="404",le="0.05"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-69fbf79577-d5q2t",host="testapp.local",ingress="test-ingress",method="GET",namespace="test",path="/",service="",status="404",le="0.1"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-69fbf79577-d5q2t",host="testapp.local",ingress="test-ingress",method="GET",namespace="test",path="/",service="",status="404",le="0.2"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-69fbf79577-d5q2t",host="testapp.local",ingress="test-ingress",method="GET",namespace="test",path="/",service="",status="404",le="0.3"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-69fbf79577-d5q2t",host="testapp.local",ingress="test-ingress",method="GET",namespace="test",path="/",service="",status="404",le="0.5"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-69fbf79577-d5q2t",host="testapp.local",ingress="test-ingress",method="GET",namespace="test",path="/",service="",status="404",le="1"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-69fbf79577-d5q2t",host="testapp.local",ingress="test-ingress",method="GET",namespace="test",path="/",service="",status="404",le="5"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-69fbf79577-d5q2t",host="testapp.local",ingress="test-ingress",method="GET",namespace="test",path="/",service="",status="404",le="10"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-69fbf79577-d5q2t",host="testapp.local",ingress="test-ingress",method="GET",namespace="test",path="/",service="",status="404",le="+Inf"} 5

Checklist:

My change requires a change to the documentation.
I have updated the documentation accordingly.
I've read the CONTRIBUTION guide
I have added tests to cover my changes.
All new and existing tests passed.

k8s-ci-robot · 2021-05-27T07:58:45Z

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
If you have done the above and are still having issues with the CLA being reported as unsigned, please log a ticket with the Linux Foundation Helpdesk: https://support.linuxfoundation.org/
Should you encounter any issues with the Linux Foundation Helpdesk, send a message to the backup e-mail support address at: [email protected]

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

k8s-ci-robot · 2021-05-27T07:58:49Z

Welcome @serge-r!

It looks like this is your first PR to kubernetes/ingress-nginx 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/ingress-nginx has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2021-05-27T07:58:50Z

Hi @serge-r. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

serge-r · 2021-05-27T08:48:36Z

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.

If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.

If you have done the above and are still having issues with the CLA being reported as unsigned, please log a ticket with the Linux Foundation Helpdesk: https://support.linuxfoundation.org/

Should you encounter any issues with the Linux Foundation Helpdesk, send a message to the backup e-mail support address at: [email protected]

/check-cla

tokers · 2021-05-27T10:56:01Z

cmd/nginx/flags.go

 	"flag"
 	"fmt"
+	"github.com/spf13/pflag"


It would be better to keep 3rd party modules after the standard ones.

Thank you. Fixed

tokers · 2021-05-27T10:57:13Z

internal/ingress/metric/collectors/socket.go

@@ -98,9 +100,11 @@ var (
 // updating the library to latest version changed the output of the metrics
 var defObjectives = map[float64]float64{0.5: 0.05, 0.9: 0.01, 0.99: 0.001}

+//var defBuckets = []float64{0.01, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 1.5, 2.0, 10.0}


I think we can remove this? If we no longer use it.

serge-r · 2021-06-04T05:50:18Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: serge-r
To complete the pull request process, please assign elvinefendi after the PR has been reviewed.
You can assign the PR to them by writing /assign @elvinefendi in a comment when ready.

/assign @ElvinEfendi

serge-r · 2021-06-22T05:59:31Z

@tokers @ElvinEfendi @rikatz Hello guys, any updates?

iamNoah1 · 2021-08-03T15:01:31Z

@serge-r reminder. PR needs a rebase

kd7lxl · 2021-10-12T16:12:11Z

/assign @kd7lxl

kd7lxl · 2021-10-12T16:23:19Z

cmd/nginx/flags.go

@@ -157,6 +159,7 @@ Requires the update-status parameter.`)
 			`Enables the collection of NGINX metrics`)
 		metricsPerHost = flags.Bool("metrics-per-host", true,
 			`Export metrics per-host`)
+		metricsBucketStr    = flags.String("metrics-buckets", "[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5, 10]", "Set of buckets which will be used for prometheus histogram metrics")


Since this is using pflag, the native type for this would be Float64Slice(). DurationSlice() might also be appropriate. This way the library handles the parsing and you won't need to unmarshal json.

kd7lxl · 2021-10-12T16:23:25Z

cmd/nginx/flags.go

@@ -157,6 +159,7 @@ Requires the update-status parameter.`)
 			`Enables the collection of NGINX metrics`)
 		metricsPerHost = flags.Bool("metrics-per-host", true,
 			`Export metrics per-host`)
+		metricsBucketStr    = flags.String("metrics-buckets", "[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5, 10]", "Set of buckets which will be used for prometheus histogram metrics")


You might consider referencing prometheus.DefBuckets explicitly as the default instead of duplicating it.

strongjz · 2021-11-09T17:44:31Z

@serge-r any update on the comments for this review?

serge-r · 2021-11-12T08:25:47Z

Hello. Thank you for review and sorry for delay. I will check and fix on next week.

k8s-ci-robot · 2021-11-29T15:53:30Z

@kd7lxl: changing LGTM is restricted to collaborators

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

iamNoah1

/ok-to-test

rikatz · 2021-12-17T21:55:11Z

/assign

iamNoah1 · 2022-01-12T18:43:17Z

@kd7lxl you will need to rebase in order to get the sec issues fixed :)

kd7lxl · 2022-01-12T18:43:59Z

@kd7lxl you will need to rebase in order to get the sec issues fixed :)

Sorry, I don't think I have permission to rebase @serge-r's fork.

iamNoah1 · 2022-01-12T19:12:28Z

ups sorry I meant @serge-r

rikatz · 2022-01-12T21:54:28Z

Hi,

First of all, sorry for the long delay. It has been busy days, and I completely missed this PR. Next time you need my attention, please feel free to ping me in Slack.

I'll approve this, but you need to rebase over main again (sorry...) so the gosec issue is gone. This was a local bug here :/

/approve
/lgtm
/hold

Thanks!

rikatz · 2022-01-15T01:25:36Z

/lgtm
Thanks

rikatz · 2022-01-15T01:25:53Z

/hold cancel

k8s-ci-robot · 2022-01-15T01:26:13Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kd7lxl, rikatz, serge-r

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [rikatz]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Crevil · 2022-01-19T09:39:53Z

Thank you so much for impementing this @iamNoah1 🙌

k8s-ci-robot added the cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. label May 27, 2021

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label May 27, 2021

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label May 27, 2021

k8s-ci-robot requested review from ElvinEfendi and rikatz May 27, 2021 07:59

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels May 27, 2021

tokers reviewed May 27, 2021

View reviewed changes

k8s-ci-robot assigned ElvinEfendi Jun 4, 2021

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 24, 2021

rikatz changed the base branch from master to main July 16, 2021 13:12

serge-r force-pushed the histogram-extend branch from deb749f to 660dbb7 Compare August 5, 2021 01:47

serge-r force-pushed the histogram-extend branch from 660dbb7 to 4ec91b8 Compare October 1, 2021 11:48

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 1, 2021

k8s-ci-robot assigned kd7lxl Oct 12, 2021

kd7lxl reviewed Oct 12, 2021

View reviewed changes

iamNoah1 reviewed Dec 2, 2021

View reviewed changes

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Dec 2, 2021

serge-r force-pushed the histogram-extend branch from 29526d4 to 17025a3 Compare December 14, 2021 11:09

k8s-ci-robot assigned rikatz Dec 17, 2021

kd7lxl mentioned this pull request Jan 12, 2022

Document prometheus metrics #2924

Closed

k8s-ci-robot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. lgtm "Looks good to me", indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Jan 12, 2022

AndrewFarley mentioned this pull request Jan 13, 2022

Adding higher latency (30,60) second buckets for Prometheus #8132

Closed

Add ability to use custom prometheus buckets

5164275

serge-r force-pushed the histogram-extend branch from 17025a3 to 5164275 Compare January 14, 2022 07:12

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 14, 2022

k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. and removed do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels Jan 15, 2022

k8s-ci-robot merged commit c0098f3 into kubernetes:main Jan 15, 2022

iamNoah1 mentioned this pull request Jan 19, 2022

[Feature request] Configure prometheus response time buckets #3898

Closed

nlybe mentioned this pull request Feb 10, 2022

Unknown flag "size-buckets" #8233

Closed

tao12345666333 mentioned this pull request Feb 27, 2022

v1.1.2 Release #8274

Closed

rchshld pushed a commit to joomcode/ingress-nginx that referenced this pull request May 19, 2023

Add ability to use custom prometheus buckets (kubernetes#7171)

d1b1a52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add flag to set custom buckets for prometheus histogram #7171

Add flag to set custom buckets for prometheus histogram #7171

serge-r commented May 27, 2021 •

edited

Loading

k8s-ci-robot commented May 27, 2021

k8s-ci-robot commented May 27, 2021

k8s-ci-robot commented May 27, 2021

serge-r commented May 27, 2021

tokers May 27, 2021

serge-r May 27, 2021

tokers May 27, 2021

serge-r May 27, 2021

serge-r commented Jun 4, 2021

serge-r commented Jun 22, 2021

iamNoah1 commented Aug 3, 2021

kd7lxl commented Oct 12, 2021

kd7lxl Oct 12, 2021 •

edited

Loading

kd7lxl Oct 12, 2021

strongjz commented Nov 9, 2021

serge-r commented Nov 12, 2021

k8s-ci-robot commented Nov 29, 2021

iamNoah1 left a comment

rikatz commented Dec 17, 2021

iamNoah1 commented Jan 12, 2022

kd7lxl commented Jan 12, 2022

iamNoah1 commented Jan 12, 2022

rikatz commented Jan 12, 2022

rikatz commented Jan 15, 2022

rikatz commented Jan 15, 2022

k8s-ci-robot commented Jan 15, 2022

Crevil commented Jan 19, 2022

Add flag to set custom buckets for prometheus histogram #7171

Add flag to set custom buckets for prometheus histogram #7171

Conversation

serge-r commented May 27, 2021 • edited Loading

What this PR does / why we need it:

Types of changes

Which issue/s this PR fixes

How Has This Been Tested?

Checklist:

k8s-ci-robot commented May 27, 2021

k8s-ci-robot commented May 27, 2021

k8s-ci-robot commented May 27, 2021

serge-r commented May 27, 2021

tokers May 27, 2021

Choose a reason for hiding this comment

serge-r May 27, 2021

Choose a reason for hiding this comment

tokers May 27, 2021

Choose a reason for hiding this comment

serge-r May 27, 2021

Choose a reason for hiding this comment

serge-r commented Jun 4, 2021

serge-r commented Jun 22, 2021

iamNoah1 commented Aug 3, 2021

kd7lxl commented Oct 12, 2021

kd7lxl Oct 12, 2021 • edited Loading

Choose a reason for hiding this comment

kd7lxl Oct 12, 2021

Choose a reason for hiding this comment

strongjz commented Nov 9, 2021

serge-r commented Nov 12, 2021

k8s-ci-robot commented Nov 29, 2021

iamNoah1 left a comment

Choose a reason for hiding this comment

rikatz commented Dec 17, 2021

iamNoah1 commented Jan 12, 2022

kd7lxl commented Jan 12, 2022

iamNoah1 commented Jan 12, 2022

rikatz commented Jan 12, 2022

rikatz commented Jan 15, 2022

rikatz commented Jan 15, 2022

k8s-ci-robot commented Jan 15, 2022

Crevil commented Jan 19, 2022

serge-r commented May 27, 2021 •

edited

Loading

kd7lxl Oct 12, 2021 •

edited

Loading