Etcd-Backup-Restore

Etcd-backup-restore is collection of components to backup and restore the etcd. It also, provides the ability to validate the data directory, so that we could know the data directory is in good shape to bootstrap etcd successfully.

Table of Contents

• Getting started
  • Prerequisites
  • Build
• Design
• Usage
  • Cloud Provider Credentials
  • Taking scheduled snapshot
  • Etcd data directory initialization
  • Etcdbrctl server
• Dependency management
  • Updating dependencies
• Testing

Getting started

Currently there are no binary build available, but it is pretty straight forward to build it by following the steps mentioned below.

Prerequisites

Although the following installation instructions are for Mac OS X, similar alternate commands could be found for any Linux distribution

Installing Golang environment

Install the latest version of Golang (at least v1.9.4 is required). For Mac OS, you could use Homebrew:

brew install golang

For other OS, please check Go installation documentation.

Make sure to set your $GOPATH environment variable properly (conventionally, it points to $HOME/go).

For your convenience, you can add the bin directory of the $GOPATH to your $PATH: PATH=$PATH:$GOPATH/bin, but it is not necessarily required.

We use Dep for managing golang package dependencies. Please install it on Mac OS via

brew install dep

On other operating systems, please check the Dep installation documentation and the Dep releases page. After downloading the appropriate release in your $GOPATH/bin folder, you need to make it executable via chmod +x <dep-release> and rename it to dep via mv dep-<release> dep.

Golint

In order to perform linting on the Go source code, please install Golint:

go get -u github.com/golang/lint/golint

Ginkgo and Gomega

In order to perform tests on the Go source code, please install Ginkgo and Gomega. Please make yourself familiar with both frameworks and read their introductions after installation:

go get -u github.com/onsi/ginkgo/ginkgo
go get -u github.com/onsi/gomega

Installing git

We use git as VCS which you would need to install.

On Mac OS run

brew install git

Installing gcloud SDK (Optional)

In case you have to create a new release or a new hotfix, you have to push the resulting Docker image into a Docker registry. Currently, we are using the Google Container Registry (this could change in the future). Please follow the official installation instructions from Google.

Installing Docker (Optional)

In case you want to build Docker images, you have to install Docker itself. We recommend using Docker for Mac OS X which can be downloaded from here.

Build

First, you need to create a target folder structure before cloning and building etcdbrctl.

mkdir -p ~/go/src/github.com/gardener
cd ~/go/src/github.com/gardener
git clone [email protected]:gardener/etcd-backup-restore.git
cd etcd-backup-restore

To build the binary in local machine environment, use make target build-local.

make build-local

This will build the binary etcdbrctl under bin directory.

Next you can make it available to use as shell command by moving the executable to /usr/local/bin.

Design

Please find the design doc here.

Usage

You can follow the help flag on etcdbrctl command and its sub-commands to know the usage details. Some of the common use cases are mentioned below. Although examples below uses AWS S3 as storage provider, we have added support for AWS, GCS, Azure and Openstack swift object store. It also supports local disk as storage provider.

Cloud Provider Credentials

The procedure to provide credentials to access the cloud provider object store varies for different providers.

For AWS S3, the credentials file has to be provided in the ~/.aws directory.

For GCP Containers, the service account json file should be provided in the ~/.gcp as a service-account-file.json file.

For Azure Blob storage, STORAGE_ACCOUNT and STORAGE_KEY should be made available as environment variables.

For Openstack Swift, OS_USERNAME, OS_PASSWORD, OS_AUTH_URL, OS_TENANT_ID and OS_DOMAIN_ID should be made available as environment variables.

Taking scheduled snapshot

etcd should already be running. One can apply standard cron format scheduling for regular backup of etcd. The cron schedule is used to take full backups. The delta snapshots are taken at regular intervals in the period in between full snapshots as indicated by the delta-snapshot-period-seconds flag. The default for the same is 10 seconds.

etcd-backup-restore has two garbage collection policies to collect existing backups from the cloud bucket. The flag garbage-collection-policy is used to indicate the correct garbage collection policy.

1. Exponential
2. LimitBased

If using LimitBased policy, the max-backups flag should be provided to indicate the number of recent backups to persist at each garbage collection cycle.

$ ./bin/etcdbrctl snapshot --storage-provider="S3" --etcd-endpoints http://localhost:2379 --schedule "*/1 * * * *" --store-container="etcd-backup" --delta-snapshot-period-seconds=10 --max-backups=10 --garbage-collection-policy='LimitBased'
INFO[0000] Validating schedule...
INFO[0000] Job attempt: 1
INFO[0000] Taking initial full snapshot at time: 2018-07-09 12:09:04.3567024 +0000 UTC
INFO[0000] Successfully opened snapshot reader on etcd
INFO[0000] Successfully saved full snapshot at: Backup-1531138145/Full-00000000-00000001-1531138145
INFO[0000] Will take next full snapshot at time: 2018-07-09 12:10:00 +0000 UTC
INFO[0000] Applied watch on etcd from revision: 00000002
INFO[0000] No events received to save snapshot.

The command mentioned above takes hourly snapshots and pushs it to S3 bucket named "etcd-backup". It is configured to keep only last 10 backups in bucket.

Exponential policy stores the snapshots in a condensed manner as mentioned below:

• All full backups and delta backups for the previous hour.
• Latest full snapshot of each previous hour for the day.
• Latest full snapshot of each previous day for 7 days.
• Latest full snapshot of the previous 4 weeks.

$ ./bin/etcdbrctl snapshot --storage-provider="S3" --etcd-endpoints http://localhost:2379 --schedule "*/1 * * * *" --store-container="etcd-backup" --delta-snapshot-period-seconds=10 --garbage-collection-policy='Exponential'
INFO[0000] Validating schedule...
INFO[0000] Job attempt: 1
INFO[0000] Taking initial full snapshot at time: 2018-07-09 12:09:04.3567024 +0000 UTC
INFO[0000] Successfully opened snapshot reader on etcd
INFO[0000] Successfully saved full snapshot at: Backup-1531138145/Full-00000000-00000001-1531138145
INFO[0000] Will take next full snapshot at time: 2018-07-09 12:10:00 +0000 UTC
INFO[0000] Applied watch on etcd from revision: 00000002
INFO[0000] No events received to save snapshot.

The command mentioned above stores etcd snapshots as per the exponential policy mentioned above.

Etcd data directory initialization

Sub-command initialize does the task of data directory validation. If the data directory is found to be corrupt, the controller will restore it from the latest snapshot in the cloud store. It restores the full snapshot first and then incrementally applies the delta snapshots.

$ ./bin/etcdbrctl initialize --storage-provider="S3" --store-container="etcd-backup" --data-dir="default.etcd"
INFO[0000] Checking for data directory structure validity...
INFO[0000] Checking for data directory files corruption...
INFO[0000] Verifying snap directory...
Verifying Snapfile default.etcd/member/snap/0000000000000001-0000000000000001.snap.
INFO[0000] Verifying WAL directory...
INFO[0000] Verifying DB file...
INFO[0000] Data directory corrupt. Invalid db files: invalid database
INFO[0000] Removing data directory(default.etcd) for snapshot restoration.
INFO[0000] Finding latest snapshot...
INFO[0000] Restoring from latest snapshot: Full-00000000-00040010-1522152360...
2018-03-27 17:38:06.617280 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32
INFO[0000] Successfully restored the etcd data directory.

Etcdbrctl server

With sub-command server you can start a http server which exposes an endpoint to initialize etcd over REST interface. The server also keeps on backup schedule thread running to have periodic backups. This is mainly made available to manage an etcd instance running in a Kubernetes cluster. You can deploy the example manifest on a Kubernetes cluster to have an fault resilient etcd instance.

Dependency management

We use Dep to manage golang dependencies.. In order to add a new package dependency to the project, you can perform dep ensure -add <PACKAGE> or edit the Gopkg.toml file and append the package along with the version you want to use as a new [[constraint]].

Updating dependencies

The Makefile contains a rule called revendor which performs a dep ensure -update and a dep prune command. This updates all the dependencies to its latest versions (respecting the constraints specified in the Gopkg.toml file). The command also installs the packages which do not already exist in the vendor folder but are specified in the Gopkg.toml (in case you have added new ones).

make revendor

The dependencies are installed into the vendor folder which should be added to the VCS.

⚠️ Make sure you test the code after you have updated the dependencies!

Testing

We have created make target verify which will internally run different rule like fmt for formatting, lint for linting check and most importantly test which will check the code against predefined unit tests. Although, currently there are not enough test cases written to cover entire code, hence one should check for failure cases manually before raising pull request. We will eventually add the test cases for complete code coverage.

make verify

By default, we try to run test in parallel without computing code coverage. To get the code coverage, you will have to set environment variable COVER to true. This will log the code coverage percentage at the end of test logs. Also, all cover profile files will accumulated under test/output/coverprofile.out directory. You can visualize exact code coverage using make show-coverage.