SOLR EFS Backups

[jbrown-xentity]

User Story

In order to have easy SOLR disaster recovery, data.gov admins wants EFS to have regular backups and a recovery process.

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

• GIVEN backups are occurring of EFS data for SOLR
  WHEN SOLR EFS/data is lost happens
  THEN a backup of EFS can be restored \

Background

https://docs.aws.amazon.com/efs/latest/ug/awsbackup.html

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/efs_backup_policy

Security Considerations (required)

None, all solr data is public

Sketch

Should be adding the necessary terraform to solr brokerpack for regular backups.

Should also involve a test of restoring a backup (does not have to be a full DB/SOLR)

[nickumia-reisys]

Proof that EFS Backups are enabled,

In AWS Backup

In EFS (each volume)

[FuhuXia]

The default setting is daily backups with a 35-day retention period.
35-day is more than what we need. With same amount the resources, a 6-hour backup with 7-day retention make more sense, if more frequent backup does not impact solr performance.

[nickumia-reisys]

Default settings are as follows, Backups:

• Start within 8 hours of 0500 UTC
• Complete within 7 days
• Retained for 35-days
• Run daily

Terraform does not provide native support for configuring this plan as part of EFS. However, we may be able to create a custom Backup plan that does not use the default Automatic Backups behavior. Additional testing would be necessary to configure it properly.

[FuhuXia]

Can we tell how long did it take for one backup?

[nickumia-reisys]

We can check the status of current backups and see how long backups took previously.

[nickumia-reisys]

It would seem like the backups are relatively quick, the 23GB full-production backup took ~2 hours.

[albionzeglin-gsa]

The EFS backups are sensitive to changes that occur to files during backup. We need to confirm that either SOLR is paused during backup execution, or that SOLR is not sensitive to file inconsistencies in backup restores.