Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(blooms): Remove backoff from notify planner #13506

Merged

Conversation

salvacorts
Copy link
Contributor

What this PR does / why we need it:
In #13306 we implemented a backoff mechanism to retry connecting to the builder. We added a backoff to connect to the builder to receive tasks, and another one to send back the results.

In this PR we remove the backoff to send back the result. I ran some experiments and:

  • The context of the connection is not canceled when the connection to the planner is lost.
    • This is fine as the blocks and metas are pushed to the store and later planning cycles will take them into account.
  • When the connection is broken to the planner, regardless of the backoff, it will always fail
  • In (the unlikely) case that the planner is throttling CPU, gRPC already has some internal default retry mechanism for the Send and Receive operations.

We can revisit this if we decide to make the planner tolerant to restarts and make it able to pick it up where it failed.

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • Title matches the required conventional commits format, see here
    • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • For Helm chart changes bump the Helm chart version in production/helm/loki/Chart.yaml and update production/helm/loki/CHANGELOG.md and production/helm/loki/README.md. Example PR
  • If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

@salvacorts salvacorts changed the title WIP: Remove backoff from notify planner fix(blooms): Remove backoff from notify planner Jul 12, 2024
@salvacorts salvacorts marked this pull request as ready for review July 12, 2024 11:05
@salvacorts salvacorts requested a review from a team as a code owner July 12, 2024 11:05
@salvacorts salvacorts merged commit e506995 into main Jul 15, 2024
61 checks passed
@salvacorts salvacorts deleted the salvacorts/bloom-refactor/builderloop-fail-on-planner-restart branch July 15, 2024 07:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants