Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add history purge command to minder server cli. #3976

Merged
merged 3 commits into from
Jul 25, 2024
Merged

Add history purge command to minder server cli. #3976

merged 3 commits into from
Jul 25, 2024

Commits on Jul 25, 2024

  1. Add history purge command to minder server cli. 

    This command is used to manage the life cycle of the history log. The
business requirement is to delete all records older than 30 days
maintaining the most recent one for each entity/rule pair even if
older than 30 days.

Implementing this requirement mandates the processing of the whole set
of records older than 30 days, which cannot be processed in chunks
without creating arbitrary holes in the history. As a first
approximation, the proposed implementation loads all records in RAM,
filters out the ones to keep, and issues a series of deletions of up
to 1000 records each. A test was added to keep track of the record
size. A future improvement would be to spill records to secondary
storage, where we would perform sorting and filtering, but it was
overly complex and unjustified at this point in time.

Fixes #3636
    @blkt
    blkt committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    1a8d320 View commit details
    Browse the repository at this point in the history

  2. Moved filtering logic back in the sql statement. 

    Table `latest_evaluation_statuses` tracks the latest evaluation id for
any given entity/rule pair. Adding it via left join allows us to
determine which records are not the latest ones among those older than
30 days by relying totally on the database rather than doing the
processing in application code. This also lowers a little bit the
resources necessary to process deletions.
    @blkt
    blkt committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    c5c85ea View commit details
    Browse the repository at this point in the history

  3. Improved tests.

    @blkt
    blkt committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    ab15f72 View commit details
    Browse the repository at this point in the history