Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(sharding): use without() grouping when merging avg_over_time shard results #12176

Merged
merged 2 commits into from
Jul 1, 2024

Conversation

ashwanthgoli
Copy link
Contributor

@ashwanthgoli ashwanthgoli commented Mar 12, 2024

What this PR does / why we need it:
avg_over_time expression that uses either keep or label_format stage without any aggregation fails on main with the following error: aggregation operator '"sum"' without grouping

Given these expressions contain stages that could reduce the labelset, sharding converts it to the following form sum (sum_over_time()) / sum(count_over_time()), but vector aggregation expr evaluator always expects a grouping. This pr updates the avg_over_time shard mapper to use without() grouping for generating vector agg expr when it is not explicitly set.

example query to reproduce this error:

avg_over_time({container="query-frontend"} |= `metrics.go` | logfmt | keep splits | unwrap split [$__auto])

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • CHANGELOG.md updated
    • If the change is worth mentioning in the release notes, add add-to-release-notes label
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • For Helm chart changes bump the Helm chart version in production/helm/loki/Chart.yaml and update production/helm/loki/CHANGELOG.md and production/helm/loki/README.md. Example PR
  • If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

Copy link
Contributor

@jeschkies jeschkies left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow. That one is one me. Incredible that it took so long. Could you add another test that's checking the output. Eg the equivalence test?

@chaudum chaudum added type/bug Somehing is not working as expected backport k209 labels Jul 1, 2024
@grafanabot
Copy link
Collaborator

This PR must be merged before a backport PR will be created.

1 similar comment
@grafanabot
Copy link
Collaborator

This PR must be merged before a backport PR will be created.

@ashwanthgoli ashwanthgoli merged commit eb8a363 into main Jul 1, 2024
63 checks passed
@ashwanthgoli ashwanthgoli deleted the avg-over-time-shard-noagg branch July 1, 2024 08:18
grafanabot pushed a commit that referenced this pull request Jul 1, 2024
@TennyZhuang
Copy link

It seems that the fix was not included in latest release.

loki, version 3.1.0 (branch: k207, revision: 935aee77)

Is there any workaround, either by modifying the query, or downgrade the server to a previous version?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport k209 size/M type/bug Somehing is not working as expected
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants