Skip to content

Commit

Permalink
Exclude SpawnProgressAPIHandler from latency metrics
Browse files Browse the repository at this point in the history
It's a long running connection kept open, serving progressbar
responses via
[EventSource](https://developer.mozilla.org/en-US/docs/Web/API/EventSource).
So it can't be treated as a regular HTTP request / response.

Getting rid of this unmasks more real problems in hub response
latency by removing this noise.

Ref 2i2c-org/infrastructure#2127 (comment)
  • Loading branch information
yuvipanda committed Feb 13, 2023
1 parent eb06ccb commit 1a9c24b
Showing 1 changed file with 30 additions and 2 deletions.
32 changes: 30 additions & 2 deletions dashboards/jupyterhub.jsonnet
Original file line number Diff line number Diff line change
Expand Up @@ -202,11 +202,39 @@ local hubResponseLatency = graphPanel.new(
datasource='$PROMETHEUS_DS'
).addTargets([
prometheus.target(
'histogram_quantile(0.99, sum(rate(jupyterhub_request_duration_seconds_bucket{app="jupyterhub", namespace=~"$hub"}[5m])) by (le))',
|||
histogram_quantile(
0.99,
sum(
rate(
jupyterhub_request_duration_seconds_bucket{
app="jupyterhub",
namespace=~"$hub",
# Ignore SpawnProgressAPIHandler, as it is a EventSource stream
# and keeps long lived connections open
handler!="jupyterhub.apihandlers.users.SpawnProgressAPIHandler"
}[5m]
)
) by (le))
|||,
legendFormat='99th percentile'
),
prometheus.target(
'histogram_quantile(0.50, sum(rate(jupyterhub_request_duration_seconds_bucket{app="jupyterhub", namespace=~"$hub"}[5m])) by (le))',
|||
histogram_quantile(
0.50,
sum(
rate(
jupyterhub_request_duration_seconds_bucket{
app="jupyterhub",
namespace=~"$hub",
# Ignore SpawnProgressAPIHandler, as it is a EventSource stream
# and keeps long lived connections open
handler!="jupyterhub.apihandlers.users.SpawnProgressAPIHandler"
}[5m]
)
) by (le))
|||,
legendFormat='50th percentile'
),
]);
Expand Down

0 comments on commit 1a9c24b

Please sign in to comment.