Feat/backup claduie

docs/creating-claudie-backup/creating-claudie-backup.md

@@ -0,0 +1,86 @@
+# Creating Claudie Backup
+
+In this section we'll explain where the state of Claudie is and
+backing up the necessary components and restoring them on a completely
+new cluster.
+
+## Claudie state
+
+Claudie stores its state in 3 different places. 
+
+-   Input Manifests are stored in <b>Mongo</b>.
+-   Terraform state files are stored in <b>MinIO</b>
+-   Locking Mechanism for the state files is implemented via <b>DynamoDB</b>
+
+These are the only services that will have a PVC attached to it, the other are stateless.
+
+## Backing up Claudie
+
+All resources that are deployed or created by Claudie can be identified with the following label:
+
+```
+    app.kubernetes.io/part-of: claudie
+```
+
+!!! note "If you want to include your deployed Input Manifests to be part of the backup you'll have to add the same label to them."
+
+We'll walk through the following scenario step-by-step to back up claudie and then restore it. 
+
+Claudie is already deployed on an existing Management Cluster and at least 1 Input Manifest has been applied. The state
+is backed up and the Management Cluster is replaced by a new one on which we restore the state.
+
+!!! note "To back up the resources we'll be using Velero version v1.11.0"
+
+The following steps will all be executed with the existing Management Cluster in context.
+
+1. To create a backup, Velero needs to store the state to external storage. The list of supported
+   providers for the external storage can be found in the [link](https://velero.io/docs/v1.11/supported-providers/).
+   In this guide we'll be using AWS S3 object storage for our backup.
+
+
+2. Prepare the S3 bucket by following the first two steps in this [setup guide](https://github.com/vmware-tanzu/velero-plugin-for-aws#setup), excluding the installation step, as this will be different for our use-case.
+
+
+!!! note "If you do not have the aws CLI locally installed, follow the [user guide](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) to set it up."
+
+3. Execute the following command to install Velero on the Management Cluster.
+```bash 
+velero install \
+--provider aws \
+--plugins velero/velero-plugin-for-aws:v1.6.0 \
+--bucket $BUCKET \
+--secret-file ./credentials-velero \
+--backup-location-config region=$REGION \
+--snapshot-location-config region=$REGION \
+--use-node-agent \
+--default-volumes-to-fs-backup
+```
+
+Following the instructions in step 2, you should have a `credentials-velero` file with the access and secret keys for the aws setup. The env variables `$BUCKET` and `$REGION` should be set to the name and region for the bucket created in AWS S3.
+
+By default Velero will use your default config `$HOME/.kube/config`, if this is not the config that points to your Management Cluster, you can override it with the `--kubeconfig` argument.
+
+4. Backup claudie by executing
+```bash
+velero backup create claudie-backup --selector app.kubernetes.io/part-of=claudie
+```
+
+To track the progress of the backup execute
+```bash
+velero backup describe claudie-backup --details
+```
+
+From this point the new Management Cluster for Claudie is in context.
+We expect that your default `kubeconfig` points to the new Management Cluster, if it does not, you can override it in the following commands using `--kubeconfig ./path-to-config`.
+
+1. Repeat the step to install Velero, but now on the new Management Cluster.
+2. Install cert manager to the new Management Cluster by executing:
+```bash
+kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.12.0/cert-manager.yaml
+```
+3. To restore the state that was stored in the S3 bucket execute
+```bash
+velero restore create --from-backup claudie-backup
+```
+
+Once all resources are restored, you should be able to deploy new input manifests and also modify existing infrastructure  without any problems.

manifests/claudie/crd/claudie.io_inputmanifests.yaml

@@ -6,6 +6,8 @@ metadata:
     controller-gen.kubebuilder.io/version: v0.11.3
   creationTimestamp: null
   name: inputmanifests.claudie.io
+  labels:
+    app.kubernetes.io/part-of: claudie
 spec:
   group: claudie.io
   names:

manifests/claudie/dynamo/kustomization.yaml

@@ -10,3 +10,6 @@ secretGenerator:
     files:
       - AWS_ACCESS_KEY_ID=secrets/access-key
       - AWS_SECRET_ACCESS_KEY=secrets/secret-key
+    options:
+      labels:
+        app.kubernetes.io/part-of: claudie

manifests/claudie/kustomization.yaml

@@ -52,6 +52,9 @@ configMapGenerator:
 - envs:
   - .env
   name: env
+  options:
+    labels:
+      app.kubernetes.io/part-of: claudie
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 images:
@@ -68,7 +71,7 @@ images:
 - name: ghcr.io/berops/claudie/kube-eleven
   newTag: ee4001d-2218
 - name: ghcr.io/berops/claudie/kuber
-  newTag: ee4001d-2218
+  newTag: 0addb19-2220
 - name: ghcr.io/berops/claudie/scheduler
   newTag: ee4001d-2218
 - name: ghcr.io/berops/claudie/terraformer

manifests/claudie/minio/kustomization.yaml

@@ -11,3 +11,7 @@ secretGenerator:
   files:
   - MINIO_ROOT_USER=secrets/access-key
   - MINIO_ROOT_PASSWORD=secrets/secret-key
+  options:
+    labels:
+      app.kubernetes.io/part-of: claudie
+

manifests/claudie/mongo/kustomization.yaml

@@ -7,4 +7,7 @@ secretGenerator:
   - name: mongo-secret
     files:
       - "MONGO_INITDB_ROOT_USERNAME=./secrets/username"
-      - "MONGO_INITDB_ROOT_PASSWORD=./secrets/password"
+      - "MONGO_INITDB_ROOT_PASSWORD=./secrets/password"
+    options:
+      labels:
+        app.kubernetes.io/part-of: claudie

manifests/claudie/operator.yaml

@@ -151,6 +151,9 @@ metadata:
     app.kubernetes.io/part-of: claudie    
 spec:
   secretName: claudie-webhook-certificate
+  secretTemplate:
+    labels:
+      app.kubernetes.io/part-of: claudie
   dnsNames:
   - claudie-operator.claudie
   - claudie-operator.claudie.svc

mkdocs.yml

@@ -80,6 +80,7 @@ nav:
       - Claudie v0.3: CHANGELOG/changelog-0.3.x.md
       - Claudie v0.4: CHANGELOG/changelog-0.4.x.md
   - Troubleshooting: troubleshooting/troubleshooting.md
+  - Creating Claudie Backup: creating-claudie-backup/creating-claudie-backup.md
 
 extra:
   version:

services/kuber/server/domain/utils/utils.go

@@ -21,10 +21,11 @@ func GetSecretMetadata(ci *pb.ClusterInfo, projectName string, outputType Output
 	return secret.Metadata{
 		Name: fmt.Sprintf("%s-%s", cid, outputType),
 		Labels: map[string]string{
-			"claudie.io/project":    projectName,
-			"claudie.io/cluster":    ci.Name,
-			"claudie.io/cluster-id": cid,
-			"claudie.io/output":     string(outputType),
+			"claudie.io/project":        projectName,
+			"claudie.io/cluster":        ci.Name,
+			"claudie.io/cluster-id":     cid,
+			"claudie.io/output":         string(outputType),
+			"app.kubernetes.io/part-of": "claudie",
 		},
 	}
 }