Issue with high amount of metrics

[ssciarrotta]

Hello,
we're using otel to send metrics to a Splunk Enterprise HEC.
The source is prometheus /federate endpoint as indicated https://github.com/signalfx/splunk-otel-collector/tree/main/examples/prometheus-federation
solution.
It works pretty good till metrics are under about 100000 (every 20s) but as volume increses into the splunk agent log
occur some Garbage Collector warning and the metrics seem are not exported to the HEC
Here follows an extract of the otel agent log:

2022-10-25T15:34:57.942Z        warn    internal/transaction.go:111     Failed to scrape Prometheus endpoint       {"kind": "receiver", "name": "prometheus/agent", "pipeline": "metrics", "scrape_timestamp": 1666712087119, "target_labels": "{__name__=\"up\", instance=\"prometheus-operated.prometheus-user-workload.svc.cluster.local:9090\", job=\"prometheus-custom\"}"}
2022-10-25T15:35:13.046Z        info    memorylimiterprocessor/memorylimiter.go:272     Memory usage after GC.     {"kind": "processor", "name": "memory_limiter", "pipeline": "metrics", "cur_mem_mib": 362}
2022-10-25T15:36:36.753Z        warn    internal/transaction.go:111     Failed to scrape Prometheus endpoint       {"kind": "receiver", "name": "prometheus/agent", "pipeline": "metrics", "scrape_timestamp": 1666712185760, "target_labels": "{__name__=\"up\", instance=\"prometheus-operated.prometheus-user-workload.svc.cluster.local:9090\", job=\"prometheus-custom\"}"}
2022-10-25T15:36:38.356Z        info    memorylimiterprocessor/memorylimiter.go:301     Memory usage is above soft limit. Forcing a GC.    {"kind": "processor", "name": "memory_limiter", "pipeline": "metrics", "cur_mem_mib": 1478}
2022-10-25T15:36:47.247Z        warn    internal/transaction.go:111     Failed to scrape Prometheus endpoint       {"kind": "receiver", "name": "prometheus/agent", "pipeline": "metrics", "scrape_timestamp": 1666712196753, "target_labels": "{__name__=\"up\", instance=\"prometheus-operated.prometheus-user-workload.svc.cluster.local:9090\", job=\"prometheus-custom\"}"}
2022-10-25T15:36:48.759Z        info    memorylimiterprocessor/memorylimiter.go:272     Memory usage after GC.     {"kind": "processor", "name": "memory_limiter", "pipeline": "metrics", "cur_mem_mib": 425}
2022-10-25T15:36:57.563Z        warn    internal/transaction.go:111     Failed to scrape Prometheus endpoint       {"kind": "receiver", "name": "prometheus/agent", "pipeline": "metrics", "scrape_timestamp": 1666712207248, "target_labels": "{__name__=\"up\", instance=\"prometheus-operated.prometheus-user-workload.svc.cluster.local:9090\", job=\"prometheus-custom\"}"}
2022-10-25T15:37:17.254Z        warn    internal/transaction.go:111     Failed to scrape Prometheus endpoint       {"kind": "receiver", "name": "prometheus/agent", "pipeline": "metrics", "scrape_timestamp": 1666712227119, "target_labels": "{__name__=\"up\", instance=\"prometheus-operated.prometheus-user-workload.svc.cluster.local:9090\", job=\"prometheus-custom\"}"}

Currently we have assigned 2GB of memory to splunk agent.

Can you help us ?

[atoulme]

How did you assign memory? Did you configure the memory ballast of the collector or assign memory to its pod?
Have you tried to increase memory to check if the issue is indeed related to memory use?

[ssciarrotta]

Hello, we've deployed splunk otel collector agent on Openshift v4.9 through the helm chart provided here: https://github.com/signalfx/splunk-otel-collector-chart.
We've assigned memory by modifing this line:
https://github.com/signalfx/splunk-otel-collector-chart/blob/main/helm-charts/splunk-otel-collector/values.yaml#L350.
We've tried to pass from 500Mi up to 2Gi and with 0.59.0, 0.61.0, 0.62.0 releases.
In order to reproduce this issue, a 'dummy-metrics' namespace has been created, which produces by itself about 200K of different metrics (other namespaces are producing a small amount of metrics).
By configuring our receiver like this:

prometheus/agent:
      config:
        scrape_configs:
        - job_name: prometheus-custom
            scrape_interval: 20s
            params:
              'match[]':
                  - '{namespace=~".+"}'
            metrics_path: "/federate"
            static_configs:
            - targets:
              - prometheus-operated.prometheus-user-workload.svc.cluster.local:9090

it doesn't work but, if we change the config like this:

prometheus/agent:
      config:
        scrape_configs:
        - job_name: prometheus-custom
            scrape_interval: 20s
            params:
              'match[]':
                  - '{namespace!~"dummy-metrics"}'
            metrics_path: "/federate"
            static_configs:
            - targets:
              - prometheus-operated.prometheus-user-workload.svc.cluster.local:9090

it works fine (i.e. just by excluding dummy-metrics namespace; it also works if we decrease the amount of metrics produced by dummy-metrics namespace to 100K).
Thank you!

[atoulme]

It looks like these instructions https://github.com/signalfx/splunk-otel-collector-chart/blob/main/docs/troubleshooting.md and this sizing guide might help you: https://github.com/signalfx/splunk-otel-collector-chart/blob/main/docs/troubleshooting.md