[Feature request] Configure prometheus response time buckets

[JorritSalverda]

FEATURE REQUEST

Currently the buckets in metric timeline serie nginx_ingress_controller_response_duration_seconds_bucket are using the Prometheus default buckets, set at

ingress-nginx/images/custom-error-pages/metrics.go

Line 43 in 29c5d77

Buckets: append([]float64{.001, .003}, prometheus.DefBuckets...),

The default buckets are

DefBuckets = []float64{.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10}

This works for a lot of apis and websites, however we're running a number of very slow apis that have response times well above 10 seconds. With these buckets it seems they all take 10 seconds to respond, while it's actually much longer.

It would be nice to be able to configure the actually used buckets in the configmap (or per ingress if possible, but I think the current histogram is global.

For validation you might want to check if the list increases in value and is limited to a maximum number to keep the label cardinality within reasonable limits.

[aledbf]

@JorritSalverda before adding this feature we need to make histograms optional. I hope I can spend time in the next weeks to refactor the metrics to allow this

[JorritSalverda]

Cool stuff! Thx

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[tklovett]

This feature is critical in order to use NGINX Ingress request duration metrics to monitor one of our high-latency services. This service frequently takes >10s to respond, and that is normal and acceptable. Currently, NGINX latency metrics for this service our unusable.

[tklovett]

/remove-lifecycle stale

please

[Crevil]

We are currently hitting this limit as well. Is there anything we can do to help moving this forward? I can see some work have been done since @aledbf posted the initial response to this, but I'm unsure whether these changes are what he implied?

Any way, anything that we can do, let me know. We'd love to help move this.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[Crevil]

/remove-lifecycle stale

As stated before we are happy to provide the implementation for this if it would help moving this forward.

[zigmund]

We have another case. Response time of one of our services is 110 ms for p99. And almost all requests counted in 100-250ms bucket and we cannot see if it became slower to 200ms, for example.

It will be nice to have configurable buckets to add more accuracy in our case.

[trnl]

We will benefit from it as well. Now our ingress controllers each produce around 150k metrics!

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[yashbhutwala]

ping! any movement on this? this would be an awesome feature, happy to help!! 😄

[yashbhutwala]

/remove-lifecycle stale

[audip]

Hey @aledbf , I'm happy to work on it and submit a PR with the fix. Can you describe what you mean by make histograms optional? Are you referring to having a configuration option to enable/disable these histogram?

https://github.com/kubernetes/ingress-nginx/blob/master/internal/ingress/metric/collectors/socket.go#L129

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[yashbhutwala]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[freeseacher]

/remove-lifecycle stale

[sziegler-skyhook]

+1, Our organization could also use this feature.

[trnl]

At the end, we implemented the Prometheus Lua plugin and completely disabled the metrics provided by the controller itself. It's missing some things like reloads, but we can leave with it.

It's using https://github.com/knyar/nginx-lua-prometheus. Our plugin is like this:

local ngx = ngx

local _M = {}

local function convert_status(value)
    return value and value:sub(1,1) .. "xx" or "_"
end

function _M.init_worker()
    prometheus = require("plugins.prometheus.prometheus").init("prometheus_metrics")
    local buckets = {0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.75, 1, 1.5, 2.5, 5, 10, 20}

    http_requests = prometheus:counter("nginx_http_requests", "Number of HTTP requests", {"host", "namespace", "ingress", "status"})
    http_request_time = prometheus:histogram("nginx_http_request_time", "HTTP request time", {"host", "namespace", "ingress"}, buckets)
    http_request_bytes_received = prometheus:counter("nginx_http_request_bytes_received", "Number of HTTP request bytes received", {"host", "namespace", "ingress"})
    http_request_bytes_sent = prometheus:counter("nginx_http_request_bytes_sent", "Number of HTTP request bytes sent", {"host", "namespace", "ingress"})
    http_connections = prometheus:gauge("nginx_http_connections", "Number of HTTP connections", {"state"})
    http_upstream_requests = prometheus:counter("nginx_http_upstream_requests", "Number of HTTP upstream requests", {"namespace", "ingress", "service", "status"})
    http_upstream_response_time = prometheus:histogram("nginx_http_upstream_response_time", "HTTP upstream response time", {"namespace", "ingress", "service"}, buckets)
    http_upstream_header_time = prometheus:histogram("nginx_http_upstream_header_time", "HTTP upstream header time", {"namespace", "ingress", "service"}, buckets)
    http_upstream_bytes_received = prometheus:counter("nginx_http_upstream_bytes_received", "Number of HTTP upstream bytes received", {"namespace", "ingress", "service"})
    http_upstream_bytes_sent = prometheus:counter("nginx_http_upstream_bytes_sent", "Number of HTTP upstream bytes sent", {"namespace", "ingress", "service"})
    http_upstream_connect_time = prometheus:histogram("nginx_http_upstream_connect_time", "HTTP upstream connect time", {"namespace", "ingress", "service"}, {0.005, 0.01, 0.02, 0.1})
    http_upstream_up = prometheus:gauge("nginx_http_upstream_up", "Upstream peer status", {"namespace", "ingress", "service", "peer"})
end

function _M.log()
    local host = ngx.var.trace_host ~= "off" and ngx.var.server_name or "_"
    local namespace = ngx.var.namespace or "_"
    local ingress = ngx.var.ingress_name or "_"
    local service = ngx.var.service_name or "_"

    http_requests:inc(1, {host, namespace, ingress, convert_status(ngx.var.status)})
    http_request_time:observe(ngx.now() - ngx.req.start_time(), {host, namespace, ingress})
    http_request_bytes_sent:inc(tonumber(ngx.var.bytes_sent) or 0, {host, namespace, ingress})
    http_request_bytes_received:inc(tonumber(ngx.var.bytes_received) or 0, {host, namespace, ingress})

    if (ngx.var.upstream_status) then
        http_upstream_requests:inc(1, {namespace, ingress, service, convert_status(ngx.var.upstream_status)})
        http_upstream_response_time:observe(tonumber(ngx.var.upstream_response_time) or 0, {namespace, ingress, service})
        http_upstream_connect_time:observe(tonumber(ngx.var.upstream_connect_time) or 0, {namespace, ingress, service})
        http_upstream_bytes_sent:inc(tonumber(ngx.var.upstream_bytes_sent) or 0, {namespace, ingress, service})
        http_upstream_bytes_received:inc(tonumber(ngx.var.upstream_bytes_received) or 0, {namespace, ingress, service})
    end
end

return _M

You can find more details about the plugins here:
https://github.com/kubernetes/ingress-nginx/blob/master/rootfs/etc/nginx/lua/plugins/README.md

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[freeseacher]

/remove-lifecycle stale

[strongjz]

/assign @longwuyuan

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[zigmund]

/remove-lifecycle stale

[iamNoah1]

/triage accepted
/priority important-longterm

[iamNoah1]

completed with #7171

[iamNoah1]

/close

[k8s-ci-robot]

@iamNoah1: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.