Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

roachtest: require stability in rebalance by load roachtests #107834

Merged
merged 2 commits into from
Aug 2, 2023

Conversation

kvoli
Copy link
Collaborator

@kvoli kvoli commented Jul 28, 2023

The rebalance/by-load roachtests collect timeseries data from the
cluster. If these tests started quickly enough, no datapoints could
exist which would cause a out of bounds array exception.

Handle the no datapoints case, and log a message without failing the
test.

Informs: #107247


The rebalance/by-load roachtests would immediately pass once the
balance target was hit. However, it was possible that the cluster was
only transiently balanced.

Bump the timeout of all tests to be uniformly 10 minutes, from a
previous 5 minutes, require that the load remains balanced for at
least 1 minute to pass.

Informs: #107247

Release note: None

@kvoli kvoli self-assigned this Jul 28, 2023
@cockroach-teamcity
Copy link
Member

This change is Reviewable

@kvoli kvoli changed the title roachtest: wait for stability in rebalance load roachtest: require stability in rebalance by load roachtests Jul 28, 2023
The `rebalance/by-load` roachtests collect timeseries data from the
cluster. If these tests started quickly enough, no datapoints could
exist which would cause an out of bounds array error.

Handle the no datapoints case, and log a message without failing the
test.

Informs: cockroachdb#107247

Release note: None
The `rebalance/by-load` roachtests would immediately pass once the
balance target was hit. However, it was possible that the cluster was
only transiently balanced.

Bump the timeout of all tests to be uniformly 10 minutes, from a
previous 5-10 minutes, require that the load remains balanced for at
least 1 minute to pass.

Informs: cockroachdb#107247

Release note: None
@kvoli kvoli force-pushed the 230728.rebalance-by-load-stability branch from 4361aae to e1f4739 Compare July 31, 2023 13:45
@kvoli kvoli added the backport-23.1.x Flags PRs that need to be backported to 23.1 label Jul 31, 2023
@kvoli kvoli marked this pull request as ready for review July 31, 2023 18:23
@kvoli kvoli requested a review from a team as a code owner July 31, 2023 18:23
@kvoli kvoli requested review from smg260, renatolabs and andrewbaptist and removed request for a team July 31, 2023 18:23
@kvoli
Copy link
Collaborator Author

kvoli commented Jul 31, 2023

I stressed the tests for 15 runs each without failure.

Should be good to take a look.

@kvoli kvoli requested a review from sumeerbhola August 1, 2023 09:53
@andrewbaptist
Copy link
Collaborator

andrewbaptist commented Aug 1, 2023

:lgtm: Thanks for the clarification.

@kvoli
Copy link
Collaborator Author

kvoli commented Aug 1, 2023

TYFTR

bors r=andrewbaptist

@craig
Copy link
Contributor

craig bot commented Aug 1, 2023

👎 Rejected by too few approved reviews

@kvoli
Copy link
Collaborator Author

kvoli commented Aug 1, 2023

@andrewbaptist could you re-approve? I don't think it went through.

@andrewbaptist andrewbaptist self-requested a review August 1, 2023 21:49
Copy link
Collaborator

@andrewbaptist andrewbaptist left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

@kvoli
Copy link
Collaborator Author

kvoli commented Aug 1, 2023

Thanks

bors r=andrewbaptist

@craig
Copy link
Contributor

craig bot commented Aug 2, 2023

Build succeeded:

@craig craig bot merged commit f153ff2 into cockroachdb:master Aug 2, 2023
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-23.1.x Flags PRs that need to be backported to 23.1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants