Automate Permanent Deletions for Submissions after specified period

[KipSigei]

Background:
We propose implementing automatic permanent deletion of submissions from the API after a specified period, such as a month or three months. We should also provide the flexibility to configure the deletion time according to specific project requirements.

Benefits of implementing the feature/enhancement:

Data Privacy and Compliance: The automatic permanent deletion of soft deleted submissions will ensure that sensitive or outdated data is completely removed from the system, enhancing data privacy and compliance with relevant regulations such as GDPR.

Efficient Data Management: By automatically removing soft deleted submissions, the API will experience improved performance and reduced storage requirements, resulting in more efficient data management processes.

Risk Mitigation: Permanent deletion of soft deleted submissions mitigates the risk of accidental data leakage or unauthorized access. By automatically removing these submissions, chances of sensitive or PII data being exposed or misused will be significantly reduced.

Implementation Plan

Permanently Delete a specific submission instance

DELETE /api/v1/data/{pk}/{dataid}

Payload

{ "permanent_delete": "true"}

Response

HTTP 204 No Content

Permanently Delete a subset of submissions

DELETE /api/v1/data/{pk}?permanent_delete=true&instance_ids=1,2,3

Response

{
   "status_code": "200",
   "message": "3 records were deleted"
}

[kelvin-muchiri]

@KipSigei I handled a support request on Friday to soft delete all submissions of a particular form. The form had 333752 submissions that had to be deleted. This seemed such a huge amount such that the command was being killed by the pod.

The proposal is the implementation on DELETE /api/v1/data/{pk}/{dataid} needs to be asynchronous with a separate queue dedicated to deleting submissions. The current synchronous implementation will timeout if the form has lots of submissions.

[kelvin-muchiri]

@KipSigei The queryset.iterator() method did not help either. What worked was this solution but the records were too many to be deleted by a Django management command. I only managed deleting less than 1000 submissions.