[VIRTS-1970] Move files aside on --fresh instead of deleting them

[bworrell]

Description

Depends on: mitre/fieldmanual#73

When running server.py --fresh, we currently delete the files under the data directory to reset the server state. This changes the behavior to back up the files before deleting them (creating a timestamped, gzipped tarball containing the files that are about to be deleted)

Note

Every time you run the server with --fresh you will generate a new backup tarball. The backup filenames include a timestamp so you can always fine the most recent one.

Examples of directory structure pre/post --fresh (I didn't run tree -a so .gitkeep files don't appear, but they are there!):

Before:

data
├── abilities
│   ├── 7e422753-ad7a-4401-bc8b-b12a28e69c25.yml
│   └── testdir
│       └── 7e422753-ad7a-4401-bc8b-b12a28e69c25.yml
├── adversaries
│   └── 7e422753-ad7a-4401-bc8b-b12a28e69c25.yml
├── backup
├── object_store
├── objectives
│   └── 7e422753-ad7a-4401-bc8b-b12a28e69c25.yml
├── payloads
│   └── 7e422753-ad7a-4401-bc8b-b12a28e69c25.yml
├── results
│   └── 7e422753-ad7a-4401-bc8b-b12a28e69c25.yml
└── sources
    └── 7e422753-ad7a-4401-bc8b-b12a28e69c25.yml

After running w/ --fresh

data
├── abilities
├── adversaries
├── backup
│   └── backup-20210406130339.tar.gz
├── object_store
├── objectives
├── payloads
├── results
└── sources

Contents of the tarball

> tar -tzvf data/backup/backup-20210406125823.tar.gz
-rw-r--r--  0 bworrell staff     603 Apr  6 08:58 data/abilities/7e422753-ad7a-4401-bc8b-b12a28e69c25.yml
drwxr-xr-x  0 bworrell staff       0 Apr  6 08:58 data/abilities/testdir/
-rw-r--r--  0 bworrell staff     603 Apr  6 08:58 data/abilities/testdir/7e422753-ad7a-4401-bc8b-b12a28e69c25.yml
-rw-r--r--  0 bworrell staff     603 Apr  6 08:58 data/adversaries/7e422753-ad7a-4401-bc8b-b12a28e69c25.yml
-rw-r--r--  0 bworrell staff     603 Apr  6 08:58 data/objectives/7e422753-ad7a-4401-bc8b-b12a28e69c25.yml
-rw-r--r--  0 bworrell staff     603 Apr  6 08:58 data/payloads/7e422753-ad7a-4401-bc8b-b12a28e69c25.yml
-rw-r--r--  0 bworrell staff     603 Apr  6 08:58 data/results/7e422753-ad7a-4401-bc8b-b12a28e69c25.yml
-rw-r--r--  0 bworrell staff     603 Apr  6 08:58 data/sources/7e422753-ad7a-4401-bc8b-b12a28e69c25.yml
-rw-r--r--  0 bworrell staff 1137539 Apr  6 08:58 data/object_store

Why?

Sometimes users run the server with --fresh on accident and end up accidentally deleting all their progress/work. This allows users to restore their files in this case. Note that the restore procedure is not actually documented anywhere at this time. We may choose to provide some sort of --restore-previous-state flag that could load the files out of the backup directory.

Type of change

Please delete options that are not relevant.

• New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Still working on this. But I've run the server with --fresh and observed that the tarball is created as expected. Note that this does not remove the .gitkeep file under each data/* subdirectory.

I also ran the following command to restore a backup state:

> cd /path/to/caldera/repo
> tar -zxvf data/backup/backup-20210406125823.tar.gz

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works

[codecov]

Codecov Report

Merging #2101 (2bd2947) into master (58a1280) will decrease coverage by 0.05%.
The diff coverage is 33.33%.

@@            Coverage Diff             @@
##           master    #2101      +/-   ##
==========================================
- Coverage   65.68%   65.63%   -0.06%     
==========================================
  Files          65       65              
  Lines        4890     4906      +16     
==========================================
+ Hits         3212     3220       +8     
- Misses       1678     1686       +8     

Impacted Files	Coverage Δ
server.py	0.00% <0.00%> (ø)
app/service/data_svc.py	29.28% <36.00%> (+1.36%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 58a1280...2bd2947. Read the comment docs.

[nopfor]

Nit: Consider using single quotes rather than double.

[wbooth]

not blockers but I think the updated message and zipping the files would be helpful

[bworrell]

I'll need to make another fieldmanual PR to update the documentation explaining the --fresh behavior and how to restore a backup since the approach is different now.