Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Optionally ack writes to kafka on Push requests #14186

Merged
merged 8 commits into from
Sep 23, 2024

Conversation

benclive
Copy link
Contributor

What this PR does / why we need it:
Adds the ability for the distributor to send requests to multiple places (like the Tee) but also ensuring that all of those requests are successful via tracking their responses.

  • Naming: I picked "TrackedTee" as a generic name, but I'm wondering if this should be "RequiredTee" or some other name like "target" or "downstream"
  • Introduced the 'required' param which controls whether to wait for responses or if they are best-effort. Open to suggestions on where to put this - currently it lives under the ingester.kafka_ingestion Config object.

I'm not particularly sold on the re-use of the Tee model and re-using the tracker object like I have. I'm definitely open to suggestions on alternatives!

Which issue(s) this PR fixes:
Fixes https://github.com/grafana/loki-private/issues/1109

Special notes for your reviewer:
I'm wondering whether to continue with this approach and convert the existing implementation to it's own Tee so we can manage them in the same way, or if that is overkill.

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • Title matches the required conventional commits format, see here
    • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.

@benclive benclive requested a review from a team as a code owner September 19, 2024 14:09
pkg/loki/modules.go Outdated Show resolved Hide resolved
@github-actions github-actions bot added the type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories label Sep 20, 2024
return nil, fmt.Errorf("failed to start kafka client: %w", err)
}
kafkaWriter = kafka.NewProducer(kafkaClient, cfg.KafkaConfig.ProducerMaxBufferedBytes,
prometheus.WrapRegistererWithPrefix("_kafka_ingester_", registerer))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: may be kafka_distributor or just kafka ?

@cyriltovena cyriltovena self-assigned this Sep 23, 2024
Copy link
Contributor

@cyriltovena cyriltovena left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@benclive benclive merged commit 7c78232 into main Sep 23, 2024
63 checks passed
@benclive benclive deleted the benclive/optionally-make-kafka-writes-required branch September 23, 2024 10:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
size/XL type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants