Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[exporter/prometheusremotewrite] Double allocation in execute function due to snappy buffer size mismatch #34273

Closed
ben-childs-docusign opened this issue Jul 26, 2024 · 1 comment · Fixed by #34274
Labels
bug Something isn't working exporter/prometheusremotewrite needs triage New item requiring triage

Comments

@ben-childs-docusign
Copy link
Contributor

ben-childs-docusign commented Jul 26, 2024

Component(s)

exporter/prometheusremotewrite

What happened?

Description

We observed extra allocations in the execute function due to a mismatch between the buffer size passed into snappy and the buffer size that snappy needs to handle the worst case compression outcome.

Specifically this code in snappy generates a buffer size > the size of the input buffer:
https://github.com/golang/snappy/blob/master/encode.go#L24

But the buffer allocated and passed into snappy is only equal to the size of our input buffer:
https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/prometheusremotewriteexporter/exporter.go#L277

Therefore snappy ends up allocating an extra buffer and the buffer allocated by prometheus remote write is unused.

Solution is to just not pre-allocate the buffer and pass nil which is valid per the snappy documentation.

This double allocation is responsible for ~10% of overall allocations in our otel collector after fixing
#34269
dfce1b99-8998-422e-a9e6-ccc5f6401393

Steps to Reproduce

Use go tool pprof to profile memory allocations of the prometheus remote write exporter

Expected Result

Compression allocates a single buffer to handle the export

Actual Result

We allocate an extra buffer in execute which then does not get used by the snappy function.

Collector version

v0.101.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

No response

Log output

No response

Additional context

No response

@ben-childs-docusign ben-childs-docusign added bug Something isn't working needs triage New item requiring triage labels Jul 26, 2024
Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working exporter/prometheusremotewrite needs triage New item requiring triage
Projects
None yet
1 participant