-
Notifications
You must be signed in to change notification settings - Fork 896
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Increase timeout for metric purging #15312
Conversation
@jntullo unrecognized command 'bug', ignoring... Accepted commands are: add_label, assign, close_issue, move_issue, remove_label, rm_label, set_milestone |
@miq-bot add_label bug, core/queue |
add message timeout to performance metric purging
@@ -895,6 +896,7 @@ | |||
:keep_hourly_performances: 6.months | |||
:keep_realtime_performances: 4.hours | |||
:purge_window_size: 1000 | |||
:queue_timeout: 20.minutes |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do we also need to increase performance_realtime_purging_interval
to avoid overlapping in scheduler ?
config/settings.yml
Outdated
@@ -1169,7 +1169,7 @@ | |||
:orchestration_stack_retired_interval: 10.minutes | |||
:performance_collection_interval: 3.minutes | |||
:performance_collection_start_delay: 5.minutes | |||
:performance_realtime_purging_interval: 15.minutes | |||
:performance_realtime_purging_interval: 25.minutes |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
there is still 5 minutes gap between timeout and next scheduler loop, does it make sense
to make purging interval very close to timeout (may be 21.minutes
) ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@yrudman yeah, that's a good idea. I wasn't sure if we could keep the current 5 minute gap.
Checked commits jntullo/manageiq@da8d96b~...75f603c with ruby 2.2.6, rubocop 0.47.1, and haml-lint 0.20.0 |
@miq-bot add_label euwe/yes, fine/yes |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍 LGTM
Fine backport details:
|
Increase timeout for metric purging (cherry picked from commit 6271921) https://bugzilla.redhat.com/show_bug.cgi?id=1462358
@jntullo Cherry-pick results in conflicts on all files 😞 Would you mind creating a PR for Euwe? |
@simaishi will do! |
Backported to Euwe via #15702 |
BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1445498
Metric purging currently uses the default timeout of 10 minutes which is not long enough for an environment with thousands of VMs. This is causing metrics tables to grow in size so large that they eventually fill the entire VMDB.
This PR adds a
queue_timeout
to the settings formetrics_history
andperformance.history
so that it can be altered as needed, and increases it by default to "20.minutes"Thoughts @yrudman?
@miq-bot bug, core/queue