add IAM to S3 Bucket Policy in Examples.md

Altinity · Oct 21, 2024 · 6640289 · 6640289
1 parent 0581914
commit 6640289
Show file tree

Hide file tree

Showing 2 changed files with 144 additions and 68 deletions.
diff --git a/Examples.md b/Examples.md
@@ -1,5 +1,25 @@
 # Use cases of clickhouse-backup
 
+## Simple cron script for daily backups and remote upload
+
+```bash
+#!/bin/bash
+BACKUP_NAME=my_backup_$(date -u +%Y-%m-%dT%H-%M-%S)
+clickhouse-backup create $BACKUP_NAME >> /var/log/clickhouse-backup.log 2>&1
+exit_code=$?
+if [[ $exit_code != 0 ]]; then
+  echo "clickhouse-backup create $BACKUP_NAME FAILED and return $exit_code exit code"
+  exit $exit_code
+fi
+
+clickhouse-backup upload $BACKUP_NAME >> /var/log/clickhouse-backup.log 2>&1
+exit_code=$?
+if [[ $exit_code != 0 ]]; then
+  echo "clickhouse-backup upload $BACKUP_NAME FAILED and return $exit_code exit code"
+  exit $exit_code
+fi
+```
+
 ## How to convert MergeTree to ReplicatedMergeTree
 This doesn't work for tables created in `MergeTree(date_column, (primary keys columns), 8192)` format
 1. Create backup
@@ -466,6 +486,113 @@ spec:
           done
 ```
 
+## How to back up object disks to s3 with s3:CopyObject
+
+To properly make backup your object s3 disks to s3 backup bucket you need to have minimal access rights via IAM
+
+```json
+{
+  "Id": "altinity-clickhouse-backup-for-s3-iam-your-uniq-name",
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Sid": "altinity-clickhouse-backup-for-s3-iam-your-uniq-name",
+      "Action": [
+        "s3:GetBucketVersioning",
+        "s3:ListBucket"
+      ],
+      "Effect": "Allow",
+      "Resource": "arn:aws:s3:::<your-object-disks-bucket>",
+      "Principal": {
+        "AWS": [
+          "arn:aws:iam::<your-aws-acount-id-for-backup>:user/<your-backup-user>"
+        ]
+      }
+    },
+    {
+      "Sid": "altinity-clickhouse-backup-for-s3-iam-your-uniq-name",
+      "Action": [
+        "s3:DeleteObject",
+        "s3:GetObject",
+        "s3:PutObject"
+      ],
+      "Effect": "Allow",
+      "Resource": "arn:aws:s3:::<your-object-disks-bucket>/*",
+      "Principal": {
+        "AWS": [
+          "arn:aws:iam::<your-aws-acount-id-for-backup>:user/<your-backup-user>"
+        ]
+      }
+    }
+  ]
+}
+```
+Store this content into `backup.json`
+
+Use following command to detect `Principal` field value
+```
+AWS_ACCESS_KEY_ID=<backup-cretentials-access-key-id> AWS_SECRET_ACCESS_KEY=<backup-cretentials-access-secret-key> aws sts get-caller-identity
+```
+
+Use following command to put IAM policy to s3 object disks bucket
+```
+aws s3api put-bucket-policy --bucket <your-object-disk-bucket> --policy="$(cat backup.json)"  
+```
+
+## How to restore object disks to s3 with s3:CopyObject
+
+To properly restore your object s3 disks from s3 backup bucket you need to have minimal access rights via IAM
+
+
+```json
+{
+  "Id": "altinity-clickhouse-restore-for-s3-iam-your-uniq-name",
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Sid": "altinity-clickhouse-restore-for-s3-iam-your-uniq-name",
+      "Action": [
+        "s3:GetBucketVersioning",
+        "s3:ListBucket"
+      ],
+      "Effect": "Allow",
+      "Resource": "arn:aws:s3:::<your-backup-bucket>",
+      "Principal": {
+        "AWS": [
+          "arn:aws:iam::<your-aws-acount-id-for-object-disks-user>:user/<your-object-disks-user>"
+        ]
+      }
+    },
+    {
+      "Sid": "altinity-clickhouse-restore-for-s3-iam-your-uniq-name",
+      "Action": [
+        "s3:DeleteObject",
+        "s3:GetObject",
+        "s3:PutObject"
+      ],
+      "Effect": "Allow",
+      "Resource": "arn:aws:s3:::<your-backup-bucket>/*",
+      "Principal": {
+        "AWS": [
+          "arn:aws:iam::<your-aws-acount-id-for-object-disks-user>:user/<your-object-disks-user>"
+        ]
+      }
+    }
+  ]
+}
+```
+Store this content into `backup.json`
+
+Use following command to detect `Principal` field value
+```
+AWS_ACCESS_KEY_ID=<object-disks-cretentials-access-key-id> AWS_SECRET_ACCESS_KEY=<object-disks-cretentials-secret-access-key> aws sts get-caller-identity
+```
+
+Use following command to attach IAM policy to s3 object disks bucket
+```
+aws s3api put-bucket-policy --bucket <your-object-disk-bucket> --policy="$(cat backup.json)"  
+```
+
 ## How to use AWS IRSA and IAM to allow S3 backup without Explicit credentials
 
 Create Role <ROLE NAME> and IAM Policy. This field typically looks like this: 

diff --git a/ReadMe.md b/ReadMe.md
@@ -525,60 +525,25 @@ Display a list of all operations from start of API server: `curl -s localhost:71
 - Optional string query argument `filter` to filter actions on server side.
 - Optional string query argument `last` to show only the last `N` actions.
 
-## Storage types
-
-### S3
-
-In order to make backups to S3, the following permissions should be set:
-
-```json
-{
-    "Version": "2012-10-17",
-    "Statement": [
-        {
-            "Sid": "clickhouse-backup-s3-access-to-files",
-            "Effect": "Allow",
-            "Action": [
-                "s3:PutObject",
-                "s3:GetObject",
-                "s3:DeleteObject"
-            ],
-            "Resource": "arn:aws:s3:::BUCKET_NAME/*"
-        },
-        {
-            "Sid": "clickhouse-backup-s3-access-to-bucket",
-            "Effect": "Allow",
-            "Action": [
-                "s3:ListBucket",
-                "s3:GetBucketVersioning"
-            ],
-            "Resource": "arn:aws:s3:::BUCKET_NAME"
-        }
-    ]
-}
-```
-
 ## Examples
 
-### Simple cron script for daily backups and remote upload
-
-```bash
-#!/bin/bash
-BACKUP_NAME=my_backup_$(date -u +%Y-%m-%dT%H-%M-%S)
-clickhouse-backup create $BACKUP_NAME >> /var/log/clickhouse-backup.log 2>&1
-exit_code=$?
-if [[ $exit_code != 0 ]]; then
-  echo "clickhouse-backup create $BACKUP_NAME FAILED and return $exit_code exit code"
-  exit $exit_code
-fi
+- [Simple cron script for daily backups and remote upload](Examples.md#simple-cron-script-for-daily-backups-and-remote-upload)
+- [How to convert MergeTree to ReplicatedMergeTree](Examples.md#how-to-convert-mergetree-to-replicatedmergetree)
+- [How to store backups on NFS or another server](Examples.md#how-to-store-backups-on-nfs-backup-drive-or-another-server-via-sftp)
+- [How to move data to another clickhouse server](Examples.md#how-to-move-data-to-another-clickhouse-server)
+- [How to monitor that backups created and uploaded correctly](Examples.md#how-to-monitor-that-backups-were-created-and-uploaded-correctly)
+- [How to back up / restore a sharded cluster](Examples.md#how-to-back-up--restore-a-sharded-cluster)
+- [How to back up a sharded cluster with Ansible](Examples.md#how-to-back-up-a-sharded-cluster-with-ansible)
+- [How to back up a database with several terabytes of data](Examples.md#how-to-back-up-a-database-with-several-terabytes-of-data)
+- [How to use clickhouse-backup in Kubernetes](Examples.md#how-to-use-clickhouse-backup-in-kubernetes)
+- [How to back up object disks to s3 with s3:CopyObject](Examples.md#how-to-back-up-object-disks-to-s3-with-s3copyobject)
+- [How to restore object disks to s3 with s3:CopyObject](Examples.md#how-to-restore-object-disks-to-s3-with-s3copyobject)
+- [How to use AWS IRSA and IAM to allow S3 backup without Explicit credentials](Examples.md#how-to-use-aws-irsa-and-iam-to-allow-s3-backup-without-explicit-credentials)
+- [How to do incremental backups work to remote storage](Examples.md#how-incremental-backups-work-with-remote-storage)
+- [How to watch backups work](Examples.md#how-to-watch-backups-work)
 
-clickhouse-backup upload $BACKUP_NAME >> /var/log/clickhouse-backup.log 2>&1
-exit_code=$?
-if [[ $exit_code != 0 ]]; then
-  echo "clickhouse-backup upload $BACKUP_NAME FAILED and return $exit_code exit code"
-  exit $exit_code
-fi
-```
+## Original Author
+Altinity wants to thank [@AlexAkulov](https://github.com/AlexAkulov) for creating this tool and for his valuable contributions.
 
 ## Common CLI Usage
 
@@ -902,20 +867,4 @@ OPTIONS:
    --full-interval value                      Interval for run 'create_remote'+'delete local' when stop create incremental backup sequence and create full backup, look format https://pkg.go.dev/time#ParseDuration
    --watch-backup-name-template value         Template for new backup name, could contain names from system.macros, {type} - full or incremental and {time:LAYOUT}, look to https://go.dev/src/time/format.go for layout examples
    
-```
-
-### More use cases of clickhouse-backup
-
-- [How to convert MergeTree to ReplicatedMergeTree](Examples.md#how-to-convert-mergetree-to-replicatedmergetree)
-- [How to store backups on NFS or another server](Examples.md#how-to-store-backups-on-nfs-backup-drive-or-another-server-via-sftp)
-- [How to move data to another clickhouse server](Examples.md#how-to-move-data-to-another-clickhouse-server)
-- [How to monitor that backups created and uploaded correctly](Examples.md#how-to-monitor-that-backups-were-created-and-uploaded-correctly)
-- [How to back up / restore a sharded cluster](Examples.md#how-to-back-up--restore-a-sharded-cluster)
-- [How to back up a sharded cluster with Ansible](Examples.md#how-to-back-up-a-sharded-cluster-with-ansible)
-- [How to back up a database with several terabytes of data](Examples.md#how-to-back-up-a-database-with-several-terabytes-of-data)
-- [How to use clickhouse-backup in Kubernetes](Examples.md#how-to-use-clickhouse-backup-in-kubernetes)
-- [How to do incremental backups work to remote storage](Examples.md#how-incremental-backups-work-with-remote-storage)
-- [How to watch backups work](Examples.md#how-to-watch-backups-work)
-
-## Original Author
-Altinity wants to thank @[AlexAkulov](https://github.com/AlexAkulov) for creating this tool and for his valuable contributions.
+```