Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

backuppb: fix incorrect ExportRequest throughput aggregation #96968

Merged
merged 1 commit into from
Feb 13, 2023

Conversation

adityamaru
Copy link
Contributor

ExportStats are emitted on the return of every ExportRequest. A tracing aggregator on each backup processor listens for these events and maintains a running aggregate. This aggregate is then used to generate interesting metrics, one of which is the throughput in MB/sec at which this processor is processing ExportRequests.

Previously, we were naively using the aggregatedBytes / aggregatedDuration to compute this thorughput. This is incorrect because we process several ExportRequests concurrently per processor. To account for this we now maintain the earliest start time and the latest end time across the requests we have aggregated. When computing throughput we use the difference between these two timestamps as our denominator.

Fixes: #89579

Release note: None

ExportStats are emitted on the return of every ExportRequest.
A tracing aggregator on each backup processor listens for these
events and maintains a running aggregate. This aggregate is then
used to generate interesting metrics, one of which is the throughput
in MB/sec at which this processor is processing ExportRequests.

Previously, we were naively using the `aggregatedBytes / aggregatedDuration`
to compute this thorughput. This is incorrect because we process
several ExportRequests concurrently per processor. To account for this
we now maintain the earliest start time and the latest end time across
the requests we have aggregated. When computing throughput we use the
difference between these two timestamps as our denominator.

Fixes: cockroachdb#89579

Release note: None
@adityamaru adityamaru requested a review from a team as a code owner February 10, 2023 18:29
@blathers-crl
Copy link

blathers-crl bot commented Feb 10, 2023

It looks like your PR touches production code but doesn't add or edit any test code. Did you consider adding tests to your PR?

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@cockroach-teamcity
Copy link
Member

This change is Reviewable

@adityamaru adityamaru changed the title backuppb: fix incorrect throughput aggregation backuppb: fix incorrect ExportRequest throughput aggregation Feb 10, 2023
@adityamaru
Copy link
Contributor Author

TFTR!

bors r=rhu713

@craig
Copy link
Contributor

craig bot commented Feb 13, 2023

Build succeeded:

@craig craig bot merged commit 8511dc0 into cockroachdb:master Feb 13, 2023
@blathers-crl
Copy link

blathers-crl bot commented Feb 13, 2023

Encountered an error creating backports. Some common things that can go wrong:

  1. The backport branch might have already existed.
  2. There was a merge conflict.
  3. The backport branch contained merge commits.

You might need to create your backport manually using the backport tool.


error creating merge commit from 602e27c to blathers/backport-release-22.2-96968: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict []

you may need to manually resolve merge conflicts with the backport tool.

Backport to branch 22.2.x failed. See errors above.


🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

backupccl: ExportStats throughput is incorrect
3 participants