Automated solution to back up etcd on a schedule from within the cluster

[SriRamanujam]

The current documentation on backing up the etcd cluster notes that there is a script on control plane nodes that can be used to take a snapshot of the etcd data and write it to to a local directory on the node. It does not provide any guidance or details on how to manage this backup process effectively. Since backups are basically the only recourse available in disaster recovery scenarios, they are very important to do correctly. Additionally, it is imperative that the backups be stored outside the cluster itself.

At the April 27th OKD Working Group meeting, there was some discussion of how to automate and manage this procedure. Several ideas were floated:

• An operator that manages a periodic cron job that would run the backup script and offsites it somehow
• A systemd unit + timer that could be dropped in via machine config
• Automating the backup of the etcd cluster to another etcd cluster

If there's community interest around one of these options (or perhaps another option entirely!) we can collaborate on putting something simple together to point people at when they ask about backups, or recommend all OKD users deploy into their clusters for peace of mind.

[vrutkovs]

Another idea worth investigating: a tekton pipeline, which can be triggered ad-hoc (or via a CronJob)

[hafe]

Also need to consider the keys for an encrypted etcd snapshot

[SriRamanujam]

I've never deployed a cluster with encrypted etcd. Are the keys dumped by the backup script along with everything else?

[duritong]

Afaik they are included in the snapshot.

[staranto]

Very much a work in progress and the codebase is a mess, but I'm working on two methods -- a systemd approach and a "k8s-native" approach. Both drive /usr/local/bin/cluster-backup.sh on a schedule and then managed the lifecycle of the snapshots. Works for my use cases, YMMV.

https://github.com/staranto/ocp4-etcd-snapshot

[SriRamanujam]

This seems like exactly the kind of thing I was thinking of! Awesome to see that someone already got around to making it :D

Which approach has worked better for you?

[danielchristianschroeter]

We manage the continuous etcd backup process with this cron one-liner on an external server. Of course this external server need ssh and optional also access via oc.
/usr/bin/ssh -i '/root/.ssh/id_rsa' -o 'StrictHostKeyChecking=no' core@/usr/local/bin/oc get node -ojsonpath='{.items[0].metadata.name}' \"/usr/bin/sudo -E /usr/bin/mkdir -p /home/core/assets/backup && /usr/bin/sudo -E /usr/bin/mount -t nfs external-nfs-backupserver.com:/okd_backup/<environment> /home/core/assets/backup && /usr/bin/sudo -E /usr/local/bin/cluster-backup.sh /home/core/assets/backup && /usr/bin/sudo -E /usr/bin/find /home/core/assets/backup -type f -mtime +6 -delete && /usr/bin/sudo -E /usr/bin/umount /home/core/assets/backup\"
Github meesed up the character escaping, here the same correctly escaped: https://pastebin.com/p1nn3W67