Add option to turn off snapshot compression

[jeffhuys]

We use snapshots internally to (relatively) quickly deploy a preview environment for a specific commit. Terraform creates a new EC2 instance and mounts a filesystem containing the most recent production snapshot. On boot, meilisearch is set to load the snapshot from the mounted drive and start running. This takes about 20-30 minutes, however, since our snapshots (compressed) weigh about 60GB.

When creating a snapshot, Meilisearch gzips the snapshot to lower the filesize. This is a time-consuming process, and while it lowers filesize, it massively increases the time it takes for our instances to come online.

My question is, would it be possible to turn off this compression, and just do a simple "copy"? This would cut down the time-to-boot on our preview environments massively.

Alternatively, I can create some hacky way of doing this (manually copying), but would much prefer an official method of doing so.

Does anyone else have experience with this / has found faster ways of doing this?

[macraig]

Hi @jeffhuys, thanks for submitting your feedback. We don't have a way of doing this currently. If we were to implement a way to snapshot an uncompressed copy of the database, would you expect it to be in a specific format? If so, which one?

[jeffhuys]

@macraig thanks for taking the time to review my suggestion!
I don't have a specific format in mind for this, but having a single file would be the most handy. We don't need to parse it manually, only copy it, and we would rather use more disk space than CPU cycles.

So, a simple .tar (if multiple actual files are being compressed) or just a big binary file are both completely fine for me! As long as it's generated relatively quickly compared to the current option; making the snapshot took 3.5 hours on our last run, and booting from it ~30 minutes, all due to the (de-)compression - the copying is the fast part.

[macraig]

Thanks @jeffhuys , we'll keep this in mind in our future planning sessions.