Unsafe code: label «run_id» of «airflow_dag_run_duration» has unlimited cardinality

[victorkashirin]

The label and metric naming guide states the following about labels:

CAUTION: Remember that every unique combination of key-value label pairs represents a new time series, which can dramatically increase the amount of data stored. Do not use labels to store dimensions with high cardinality (many different label values), such as user IDs, email addresses, or other unbounded sets of values.

Given that typical run_id includes the date of run, it should generate new time series for each individual dag run, which might eventually blow up Prometheus.

[elephantum]

@victorkashirin This is a good observation. Do you have a solution in mind?

Use case for this metric was to monitor and alert on stuck dagruns.

Maybe we do not need run_id as a dimension, but then this metric becomes ambiguous if several dagruns are executing simultaneously.

Thoughts?

[victorkashirin]

Stuck dag run will require investigation anyway, and you can retrieve information about currently running dagruns and their duration in seconds via Ad Hoc Query(/admin/queryview/) with something like this (for Postgres):

select 
  dr.run_id, 
  ROUND(EXTRACT(EPOCH FROM (now() at time zone 'utc' - dr.start_date::timestamp))) as duration
from dag_run dr
where dag_id = '<dag_id>' and state = 'running';

Here <dag_id> will be known from alert, thus it's just a matter of running this query to find which dagrun is hanging.

[elephantum]

I agree, we will change exported metrics not to include run_id