prometheus_remote_write source does not interpret empty label as null

[nzxwang]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

According to the Prometheus data model:

Labels with an empty label value are considered equivalent to labels that do not exist.

As such, when the prometheus_remote_write source receives a metric like:

{"name":"loop.prometheus.remote.write.gauge","namespace":"","tags":{"my_tag":"my_value"},"timestamp":"2024-11-03T20:35:31.897295Z","interval_ms":15000,"kind":"absolute","gauge":{"value":1.0}}

it should consider the namespace label as equivalent to not existing, but the prometheus_remote_write source instead interprets it as is and appends an _ to the name.

On a somewhat related note, I'd like to get around this by explicitly setting the namespace parameter of the sender's static_metrics component to null, but since I am using TOML and the default namespace is a stringstatic, I can't actually set this to null. Do you have any advice here?

Configuration

Sender

[sources.source_heartbeat_metric]
type = "static_metrics"
interval_secs = 15
namespace = ""

    [[sources.source_heartbeat_metric.metrics]]
    name = "loop.prometheus.remote.write.gauge"
    kind = "absolute"
    value = {gauge = {value=1.0}}

        [sources.source_heartbeat_metric.metrics.tags]
        my_tag = "my_value"

[sinks.sink_console]
type = "console"
inputs = ["source_heartbeat_metric"]
encoding.codec = "json"

[sinks.sink_prometheus_remote_write]
type = "prometheus_remote_write"
inputs = [ "source_heartbeat_metric" ]
endpoint = "http://localhost:9090"
healthcheck.enabled = false

Receiver

[sources.prometheus_remote_write_endpoint]
type = "prometheus_remote_write"
address = "0.0.0.0:9090"

[sinks.sink_console]
type = "console"
inputs = ["prometheus_remote_write_endpoint"]
encoding.codec = "json"

Version

vector 0.42.0 (x86_64-apple-darwin 3d16e34 2024-10-21 14:10:14.375255220)

Debug Output

No response

Example Data

Event Sent

{"name":"loop.prometheus.remote.write.gauge","namespace":"","tags":{"my_tag":"my_value"},"timestamp":"2024-11-03T20:43:31.896738Z","interval_ms":15000,"kind":"absolute","gauge":{"value":1.0}}

Event Received

{"name":"_loop.prometheus.remote.write.gauge","tags":{"my_tag":"my_value"},"timestamp":"2024-11-03T20:43:16.896Z","kind":"absolute","gauge":{"value":1.0}}

Additional Context

No response

References

No response

[pront]

Hi @nzxwang,

On a somewhat related note, I'd like to get around this by explicitly setting the namespace parameter of the sender's static_metrics component to null, but since I am using TOML and the default namespace is a stringstatic, I can't actually set this to null. Do you have any advice here?

This is an unfortunate limitation of TOML. We would have to change this to Option<String> defaulting to None in order to represent in TOML but then we wouldn't be able to have static as default.

Treating an empty string "" as None in static_metrics is probably the desirable fix here.

You can try the following as an immediate workaround:

[transforms.removv_namespace]
type = "remap"
inputs = ["source_heartbeat_metric"]
source = """
    if exists(.namespace) && .namespace == "" {
        del(.namespace)
    }
"""

When the events hit the sink_prometheus_remote_write, they won't have any namespace and the sink encoder won't prepend a separator.

[nzxwang]

Hi @pront thanks for your reply. I'll use the remap workaround for now and submit a followup PR for treating an empty string as None in static_metrics.