Add history purge command to minder server cli. #3976
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
blkt
force-pushed
the
issue-3636
branch
3 times, most recently
from
d3c3752
to
5d6d4ac
Compare
dmjb
reviewed
Jul 24, 2024
cmd/server/app/history_purge.go
Outdated
| return nil | ||
| } | ||
|
|
||
| // filterRecords sift through the records separating the latest for |
There was a problem hiding this comment.
I wonder if this could be implemented in the SQL query by using the latest_evaluation_statuses table and the EXCEPT operator.
There was a problem hiding this comment.
You're right, I integrated this suggestion, but I haven't used the EXCEPT operator and implemented it using joins.
blkt
force-pushed
the
issue-3636
branch
2 times, most recently
from
6ca77ea
to
d7a66c7
Compare
This command is used to manage the life cycle of the history log. The business requirement is to delete all records older than 30 days maintaining the most recent one for each entity/rule pair even if older than 30 days. Implementing this requirement mandates the processing of the whole set of records older than 30 days, which cannot be processed in chunks without creating arbitrary holes in the history. As a first approximation, the proposed implementation loads all records in RAM, filters out the ones to keep, and issues a series of deletions of up to 1000 records each. A test was added to keep track of the record size. A future improvement would be to spill records to secondary storage, where we would perform sorting and filtering, but it was overly complex and unjustified at this point in time. Fixes #3636
Table `latest_evaluation_statuses` tracks the latest evaluation id for any given entity/rule pair. Adding it via left join allows us to determine which records are not the latest ones among those older than 30 days by relying totally on the database rather than doing the processing in application code. This also lowers a little bit the resources necessary to process deletions.
dmjb
approved these changes
Jul 25, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This command is used to manage the life cycle of the history log. The business requirement is to delete all records older than 30 days maintaining the most recent one for each entity/rule pair even if older than 30 days.
Implementing this requirement mandates the processing of the whole set of records older than 30 days, which cannot be processed in chunks without creating arbitrary holes in the history. As a first approximation, the proposed implementation loads all records in RAM, filters out the ones to keep, and issues a series of deletions of up to 1000 records each. A test was added to keep track of the record size. A future improvement would be to spill records to secondary storage, where we would perform sorting and filtering, but it was overly complex and unjustified at this point in time.
Fixes #3636
Change Type
Testing
Some unit tests, mostly manual tests.
Review Checklist: