Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[output.wavefront] Introduced "immediate_flush" flag #8165

Merged
merged 5 commits into from
Nov 3, 2020
Merged

[output.wavefront] Introduced "immediate_flush" flag #8165

merged 5 commits into from
Nov 3, 2020

Conversation

prydin
Copy link
Contributor

@prydin prydin commented Sep 22, 2020

Required for all PRs:

  • Signed CLA.
  • Associated README.md updated.
  • Has appropriate unit tests.

Addresses an issue mentioned in #7409 concerning the "double buffering" done by the Wavefront SDK. The SDK is designed to send metrics asynchronously in order to minimize the delay experienced by a caller. This behavior is not ideal when used with Telegraf, since Telegraf will assume the connection to Wavefront can accept data very quickly. When the internal buffer in the Wavefront SDK overflows, it will drop metrics. Meanwhile, Telegraf will still think the metrics are flowing at a very high rate and will keep flooding the SDK.

To avoid extensive redesign of the SDK, this PR adds a flag, immediate_flush that flushes the internal SDK buffers after each batch. This will ensure that the buffer never fills with more records than the batch size and also has the side effect of providing "back pressure" to the Telegraf core. Thanks to that back pressure, Telegraf will no longer be "tricked" into thinking the connection has virtually unlimited capacity and can take whatever the action it needs to manage the throughput gracefully.

Copy link
Contributor

@ssoroka ssoroka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great, but I think there's no reason to still support the buffer or allow configuration of the flush. Keep in mind Telegraf is already doing periodic flush (which is essentially the write method) based on metric_batch_size and flush_interval.

plugins/outputs/wavefront/README.md Outdated Show resolved Hide resolved
@ssoroka ssoroka merged commit 748af7f into influxdata:master Nov 3, 2020
ssoroka pushed a commit that referenced this pull request Nov 13, 2020
arstercz pushed a commit to arstercz/telegraf that referenced this pull request Mar 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants