docs: add D&DR documentation (#122)

* docs: Initial documentation for DR SPs

* docs: Docusaurus display tweaks.

docs/app/data-retention/_category_.json

@@ -0,0 +1,5 @@
+{
+  "label": "Data Retention",
+  "position": 100,
+  "collapsed": true
+}

docs/app/data-retention/cleanup_procedure.md

@@ -0,0 +1,44 @@
+---
+sidebar_position: 2
+---
+# Delete & Cleanup Procedure
+
+The `sp_dr_cleanup` systematically cleans up data across related tables in the database, based on the data retention rules. This procedure is integral to maintaining the integrity and performance of the system by removing outdated records, and helps remain compliant with data protection and retention legislation.
+
+### Key Steps in the Cleanup Process
+
+1. **Initialization:**
+    - The procedure starts by checking system parameters to ensure that data deletion is not disabled. It sets some system parameters necessary for the run, including disabling foreign key checks and triggers for the remainder of the run.
+
+2. **Transaction Management:**
+    - A transaction is started to ensure that all changes are atomic and can be rolled back in case of any errors during the execution process. This is planned to be removed soon, to allow the DR process to run while the system remains online (albeit at a v.quiet time)
+
+3. **Temporary Tables Creation:**
+    - Temporary tables are created to manage the data that needs to be processed. These tables help isolate and track which records are to be deleted without affecting the live data during the cleanup process.
+
+4. **Deletion Criteria Setup:**
+    - Records eligible for deletion are identified based on the rules specified in the `data_retention` table and related `data_retention_rule`. This includes checking fields like `action_confirmation` and `deleted_date`.
+
+5. **Dependency Management:**
+    - The script handles dependencies carefully by ensuring that related records in dependent tables are also considered for deletion. This includes cascading effects where deleting a record in one table may affect records in multiple related tables.
+
+6. **Record Deletion:**
+    - Actual deletion of records occurs from various tables, based on the dependencies and relationships defined.
+
+7. **Verification and Cleanup:**
+    - After deletion, the script verifies the number of records deleted and ensures that the counts match the expected numbers. Temporary tables are used to compare pre and post-deletion record counts.
+
+8. **Transaction Completion:**
+    - If the `dry_run` parameter is set to `FALSE`, the changes are committed to the database. If `dry_run` is `TRUE`, all changes are rolled back, allowing the operation to be tested without affecting actual data.
+
+9. **Finalization:**
+    - The environment is reset by re-enabling foreign key checks and triggers. Temporary tables are dropped to clean up the session.
+
+### Error Handling
+
+- The procedure includes robust error handling mechanisms that catch SQL exceptions during execution. These exceptions trigger a rollback of the transaction, ensuring that the database state remains consistent and no partial updates are committed.
+- The above mechanism will not be possible when running without transactions, some consideration should be given to the impact of this change.
+
+### Usage
+
+- This procedure can be executed with parameters to specify the user ID performing the cleanup, the limit on the number of records to process, and whether the operation should be a dry run.

docs/app/data-retention/overview.md

@@ -0,0 +1,53 @@
+---
+sidebar_position: 1
+---
+# Data Retention System Overview
+
+The Data Retention system is designed to manage the lifecycle of records in compliance with regulatory and internal data management policies. This system uses a set of "populate rules" to determine when records are eligible for deletion, ensuring data is handled securely and efficiently.
+
+### Key Components
+
+#### Data Retention Rules
+
+- **Table:** `data_retention_rule`
+- **Description:** Stores the rules that define how data retention is handled for different types of entity. Each rule specifies the retention period, the maximum dataset size for consideration in a single run, and whether the rule is currently enabled.
+- **Key Fields:**
+    - `id`: Unique identifier for the rule.
+    - `description`: A brief description of what the rule does.
+    - `retention_period`: How long (in months) to keep the data after record closure.
+    - `max_data_set`: Limits the number of rows affected in a single operation.
+    - `is_enabled`: Indicates if the rule is active.
+    - `action_type`: Defines the type of action (e.g., automatic deletion, manual review).
+    - `populate_procedure`: The stored procedure associated with populating data for this rule.
+
+#### Data Retention Control Table
+
+- **Table:** `data_retention`
+- **Description:** Records information about data eligible for retention actions. It is populated by the stored procedures indexed in the  `data_retention_rule` table,
+- **Key Fields:**
+    - `entity_name`, `entity_pk`: Identify the database record to delete.
+    - `data_retention_rule_id`: Links to the applicable retention rule that populated the record.
+    - `action_confirmation`: Indicates whether the deletion action is confirmed by an admin user.
+    - `assigned_to`: User ID for accountability.
+    - `next_review_date`: When the record should next be reviewed.
+    - `actioned_date`: When the deletion took place for this record.
+
+### Process Flow
+
+1. **Rule Execution:**
+    - Stored procedures defined per rule in `populate_procedure` are executed regularly. These procedures evaluate records against their respective retention rules and populate the `data_retention` table with records eligible for deletion or review.
+
+2. **Admin UI Interaction:**
+    - An administrative UI in the `olcs-internal` web app allows users to manage retention rules and review records flagged for manual action.
+
+3. **Automated and Manual Deletion:**
+    - Records flagged for automatic deletion are processed in scheduled runs of the `sp_dr_cleanup` stored procedure, which handles the deletion of data and cleanup of any dependent records.
+    - Records requiring manual approval are not deleted in these runs until manually reviewed and confirmed through the admin UI, then deleted in subsequent cleanup runs.
+
+### Security and Compliance
+
+- The system is designed to ensure compliance with data protection regulations by allowing records to be deleted after per-entity retention periods have expired, and allows for review and manual intervention when necessary.
+
+### Conclusion
+
+The Data Retention system provides a framework for managing the lifecycle of data within the organisation. By automating data cleanup the system helps maintain data hygiene and regulatory compliance efficiently.

docs/app/data-retention/populate-procedures/_category_.json

@@ -0,0 +1,5 @@
+{
+  "label": "Populate Rules",
+  "position": 100,
+  "collapsed": true
+}

docs/app/data-retention/populate-procedures/sp_populate_admin_cancelled.md

@@ -0,0 +1,13 @@
+### sp_populate_admin_cancelled
+
+#### Parameters:
+- `p_user_id INT`: User ID of the person executing the procedure.
+
+#### Record Selection Criteria for Deletion:
+- **Entity:** `bus_reg`
+- **Status Filter:** Selects records where the `bus_reg_status_id` is `'breg_s_admin'` which is "admin cancelled" status 
+- **Date Filter:** Selects records where the `effective_date` is older than the current date minus the retention period.
+
+
+#### Summary:
+- This procedure deletes records from the `bus_reg` table where the `bus_reg_status_id` is `'breg_s_admin'` and the `effective_date` is older than the current date minus the retention period defined in the data retention rules.

docs/app/data-retention/populate-procedures/sp_populate_application_not_yet_submitted.md

@@ -0,0 +1,12 @@
+### sp_populate_application_not_yet_submitted
+
+#### Parameters:
+- `p_user_id INT`: User ID of the person executing the procedure.
+
+#### Record Selection Criteria for Deletion:
+- **Entity:** `application`
+- **Status Filter:** Selects records where the application status is `'apsts_not_submitted'`.
+- **Date Filter:** Selects records based on the `last_modified_on` or `created_on` date being older than the current date minus the retention period.
+
+#### Summary:
+- This procedure targets records in the `application` table where the status is `'apsts_not_submitted'` and either the last modified date or creation date is older than the current date minus the retention period.

docs/app/data-retention/populate-procedures/sp_populate_application_ntu_withdrawn.md

@@ -0,0 +1,13 @@
+### sp_populate_application_ntu_withdrawn
+
+#### Parameters:
+- `p_user_id INT`: User ID of the person executing the procedure.
+
+#### Record Selection Criteria for Deletion:
+- **Entity:** `application`
+- **Status Filter:** Selects records where the application status is either `'apsts_ntu'` (not taken up) or `'apsts_withdrawn'` (withdrawn by the applicant).
+- **Event Filter:** Only considers records where the last event related to the application is of type ID `5` or `16`.
+- **Date Filter:** Filters applications based on the date of the last relevant event being older than the current date minus the retention period defined.
+
+#### Summary:
+- This procedure focuses on records from the `application` table with statuses of not taken up or withdrawn, considering only those where the last significant event occurred before the current date minus the retention period.

docs/app/data-retention/populate-procedures/sp_populate_application_refused.md

@@ -0,0 +1,13 @@
+### sp_populate_application_refused
+
+#### Parameters:
+- `p_user_id INT`: User ID of the person executing the procedure.
+
+#### Record Selection Criteria for Deletion:
+- **Entity:** `application`
+- **Status Filter:** Selects records where the application status is `'apsts_refused'`.
+- **Variation Filter:** Considers only applications that are not variations (`is_variation = 0`).
+- **Date Filter:** Filters applications based on the `refused_date` being older than the current date minus the retention period.
+
+#### Summary:
+- This procedure targets records in the `application` table that have been refused and are not variations, considering those whose refusal date is older than the current date minus the retention period.

docs/app/data-retention/populate-procedures/sp_populate_cancelled_reg.md

@@ -0,0 +1,12 @@
+### sp_populate_cancelled_reg
+
+#### Parameters:
+- `p_user_id INT`: User ID of the person executing the procedure.
+
+#### Record Selection Criteria for Deletion:
+- **Entity:** `bus_reg`
+- **Status Filter:** Selects records where the bus registration status is `'breg_s_cancelled'`.
+- **Date Filter:** Filters registrations based on the `effective_date` being older than the current date minus the retention period defined for this rule.
+
+#### Summary:
+- This procedure targets records in the `bus_reg` table that have been cancelled, specifically those whose effective date is older than the current date minus the retention period.

docs/app/data-retention/populate-procedures/sp_populate_case_impounding.md

@@ -0,0 +1,11 @@
+### sp_populate_case_impounding
+
+#### Parameters:
+- `p_user_id INT`: User ID of the person executing the procedure.
+
+#### Record Selection Criteria for Deletion:
+- **Entity:** `cases`
+- **Closure Filter:** Selects case records where the `closed_date` is older than the current date minus the retention period defined for this rule.
+
+#### Summary:
+- This procedure identifies cases related to impounding that have been closed and whose closed date is older than the current date minus the retention period.

docs/app/data-retention/populate-procedures/sp_populate_document_financial_reports.md

@@ -0,0 +1,13 @@
+### sp_populate_document_financial_reports
+
+#### Parameters:
+- `p_user_id INT`: User ID of the person executing the procedure.
+
+#### Record Selection Criteria for Deletion:
+- **Entity:** `document`
+- **Sub-category Filter:** Selects records specifically in the sub-category with ID `180`, which is `Financial reports` .
+- **Association Filters:** Ensures the documents considered for deletion are not associated with any licences, applications, cases, transport managers, operating centres, IRFO organisations, submissions, statements, or continuation details.
+- **Date Filter:** Filters documents based on the last modification or creation date being older than the current date minus the retention period from the data retention rules table.
+
+#### Summary:
+- Targets documents classified under a specific financial report sub-category, which are not linked to any major entity (eg licences or applications), and whose last modification or creation date is older than the retention period.

docs/app/data-retention/populate-procedures/sp_populate_irfo_gv_permit_expired.md

@@ -0,0 +1,11 @@
+### sp_populate_irfo_gv_permit_expired
+
+#### Parameters:
+- `p_user_id INT`: User ID executing the procedure.
+
+#### Record Selection Criteria for Deletion:
+- **Entity:** `irfo_gv_permit`
+- **Expiry Filter:** Selects records where the `expiry_date` of the permit is older than the current date minus the retention period.
+
+#### Summary:
+- This procedure targets expired IRFO goods vehicle permits, populating rows where the expiry date is older than the current date minus the retention period.

docs/app/data-retention/populate-procedures/sp_populate_irfo_gv_permit_withdrawn_pending.md

@@ -0,0 +1,13 @@
+### sp_populate_irfo_gv_permit_withdrawn_pending_refused_expired
+
+#### Parameters:
+- `p_user_id INT`: User ID of the person executing the procedure.
+
+#### Record Selection Criteria for Deletion:
+- **Entity:** `irfo_gv_permit`
+- **Status Filter:** Selects records where the IRFO goods vehicle permit status is in one of the following states: 'irfo_auth_s_withdrawn' (withdrawn), 'irfo_perm_s_pending' (pending), or 'irfo_perm_s_refused' (refused).
+- **No Expiry Date Filter:** Specifically targets permits that do not have an `expiry_date` set.
+- **Creation Date Filter:** Filters permits based on the `created_on` date being older than the current date minus the retention period
+
+#### Summary:
+- This procedure identifies IRFO goods vehicle permits with specific statuses (withdrawn, pending, or refused) that lack an expiry date and whose creation date is older than the current date minus the retention period.

docs/app/data-retention/populate-procedures/sp_populate_irfo_operator_expired.md

@@ -0,0 +1,13 @@
+### sp_populate_irfo_operator_expired
+
+#### Parameters:
+- `p_user_id INT`: User ID of the person executing the procedure.
+
+#### Record Selection Criteria for Deletion:
+- **Entity:** `organisation`
+- **Type Filter:** Focuses on organisations of type 'org_t_ir', IRFO Operators.
+- **Creation Date Filter:** Selects organisations based on the `created_on` date being older than the current date minus the retention period fron the data retention rules.
+- **Association Filters:** Ensures that the organisations are not currently associated with any IRFO PSV authorizations, IRFO goods vehicle permits, or licenses.
+
+#### Summary:
+- This procedure targets IRFO operator organisations that are no longer linked to any active permits, authorizations, or licenses and whose creation date is older than the specified retention period. It stages these organization records for deletion.

docs/app/data-retention/populate-procedures/sp_populate_irfo_psv_auth_expired.md

@@ -0,0 +1,11 @@
+### sp_populate_irfo_psv_auth_expired
+
+#### Parameters:
+- `p_user_id INT`: User ID of the person executing the procedure.
+
+#### Record Selection Criteria for Deletion:
+- **Entity:** `irfo_psv_auth`
+- **Expiry Filter:** Selects records where the `expiry_date` of the PSV authorization is older than the current date minus the retention period parameter.
+
+#### Summary:
+- This procedure identifies expired IRFO PSV authorizations, selecting those whose expiry date is older than the current date minus the retention period.

docs/app/data-retention/populate-procedures/sp_populate_irfo_psv_auth_withdrawn_pending_refused_expired.md

@@ -0,0 +1,13 @@
+### sp_populate_irfo_psv_auth_withdrawn_pending_refused_expired
+
+#### Parameters:
+- `p_user_id INT`: User ID of the person executing the procedure.
+
+#### Record Selection Criteria for Deletion:
+- **Entity:** `irfo_psv_auth`
+- **Status Filter:** Selects records where the PSV authorization status is in one of the following states: 'irfo_auth_s_withdrawn' (withdrawn), 'irfo_auth_s_pending' (pending), 'irfo_auth_s_refused' (refused), 'irfo_auth_s_granted' (granted), or 'irfo_auth_s_cns' (consented).
+- **No Expiry Date Filter:** Specifically targets authorizations that do not have an `expiry_date` set.
+- **Creation Date Filter:** Filters authorizations based on the `created_on` date being older than the current date minus the retention period.
+
+#### Summary:
+- This procedure identifies IRFO PSV authorizations with specific statuses, particularly those without an expiry date and whose creation date is older than the current date minus the retention period.

docs/app/data-retention/populate-procedures/sp_populate_irhp_cancelled_withdrawn.md

@@ -0,0 +1,11 @@
+### sp_populate_irhp_cancelled_withdrawn
+
+#### Parameters:
+- `p_user_id INT`: User ID of the person executing the procedure.
+
+#### Record Selection Criteria for Deletion:
+- **Entity:** `irhp_application`
+- **Status Filter:** Selects records where the IRHP application status is either 'permit_app_cancelled' (cancelled) or 'permit_app_withdrawn' (withdrawn).
+- **Date Filter:** Filters applications based on the `last_modified_on` date being older than the current date minus the retention period.
+#### Summary:
+- This procedure identifies cancelled or withdrawn IRHP applications, selecting those whose last modification date is older than the current date minus the retention period

docs/app/data-retention/populate-procedures/sp_populate_irhp_cert_road_nys.md

@@ -0,0 +1,12 @@
+### sp_populate_irhp_cert_road_nys
+
+#### Parameters:
+- `p_user_id INT`: User ID of the person executing the procedure.
+
+#### Record Selection Criteria for Deletion:
+- **Entity:** `irhp_application`
+- **Status Filter:** Selects records where the IRHP application status is 'permit_app_nys' (not yet started).
+- **Permit Type Filter:** Specifically targets applications for permit types with IDs `6` (Roadworthiness Cert Vehicle) and `7` (Roadworthiness Cert Trailer).
+- **Date Filter:** Filters applications based on either the `last_modified_on` date or the `created_on` date being older than 12 months from the current date. This accounts for cases where the last modified date might be null.
+#### Summary:
+- This procedure identifies IRHP applications that are not yet started (status 'permit_app_nys') for specific permit types, selecting those whose last interaction or creation is older than 12 months.

docs/app/data-retention/populate-procedures/sp_populate_irhp_expired.md

@@ -0,0 +1,14 @@
+### sp_populate_irhp_expired
+
+#### Parameters:
+- `p_user_id INT`: User ID of the person executing the procedure.
+
+#### Record Selection Criteria for Deletion:
+- **Entity:** `irhp_permit` and `irhp_application`
+- **Status Filter for IRHP Permit:** Selects records where the IRHP permit status is 'irhp_permit_expired'.
+- **Status Filter for IRHP Application:** Selects records where the IRHP application status is 'permit_app_expired'.
+- **Date Filter for IRHP Permit:** Filters permits based on the `last_modified_on` date being older than the current date minus the retention window.
+- **Date Filter for IRHP Application:** Filters applications based on the `last_modified_on` date being older than 60 months from the current date.
+
+#### Summary:
+- This procedure identifies expired IRHP permits and applications, selecting those permits whose last modification date is older than the defined retention period and applications whose last modification date is older than 60 months.

docs/app/data-retention/populate-procedures/sp_populate_irhp_replaced_permits.md

@@ -0,0 +1,11 @@
+### sp_populate_irhp_replaced_permits
+
+#### Parameters:
+- `p_user_id INT`: User ID of the person executing the procedure.
+
+#### Record Selection Criteria for Deletion:
+- **Entity:** `irhp_permit`
+- **Status Filter:** Selects records where the IRHP permit status is 'irhp_permit_ceased'.
+- **Expiry Date Filter:** Filters permits based on the `expiry_date`  being earlier than the current date minus the retention period.
+#### Summary:
+- This procedure targets IRHP permits that have been ceased or replaced, selecting those whose expiry date is older than the defined retention period.

docs/app/data-retention/populate-procedures/sp_populate_irhp_terminated_permits.md

@@ -0,0 +1,13 @@
+### sp_populate_irhp_terminated_permits
+
+#### Parameters:
+- `p_user_id INT`: User ID of the person executing the procedure.
+
+#### Record Selection Criteria for Deletion:
+- **Entity:** `irhp_permit` and `irhp_application`
+- **Status Filter for IRHP Permit:** Selects records where the IRHP permit status is 'irhp_permit_terminated'.
+- **Date Filter for IRHP Permit:** Filters permits based on the `last_modified_on` date being older than the current date minus the retention period defined in the data retention rules.
+- **Comprehensive Application Cleanup:** Also targets IRHP applications associated solely with terminated permits, ensuring applications that no longer have any active or non-terminated permits and were last modified more than 60 months ago are selected.
+
+#### Summary:
+- This procedure identifies terminated IRHP permits and their associated applications, selecting those permits and applications whose last modification dates are older than the defined retention periods.