Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add documentation for configuring Kube GCS logging. #4622

Merged
merged 4 commits into from
Jul 9, 2021
Merged
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 42 additions & 8 deletions docs/deploying-airbyte/on-kubernetes.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,10 @@ Configure `kubectl` to connect to your cluster by using `kubectl use-context my-
Both `dev` and `stable` versions of Airbyte include a stand-alone `Minio` deployment. Airbyte publishes logs to this `Minio` deployment by default.
This means Airbyte comes as a **self-contained Kubernetes deployment - no other configuration is required**.

Airbyte currently supports logging to `Minio` or `S3`. The following instructions are for users wishing to log to their own `Minio` layer or `S3` bucket.
Airbyte currently supports logging to `Minio`, `S3` or `GCS`. The following instructions are for users wishing to log to their own `Minio` layer, `S3` bucket
or `GCS` bucket.

The provided credentials require both read and write permissions. The logger attempts to create the log bucket if it does not exist.

#### Configuring Custom Minio Log Location
Replace the following variables in the `.env` file in the `kube/overlays/stable` directory:
Expand Down Expand Up @@ -80,9 +83,39 @@ S3_MINIO_ENDPOINT=
S3_PATH_STYLE_ACCESS=
```

The provided credentials require both S3 read/write permissions. The logger attempts to create the bucket if it does not exist. See [here](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html)
for instructions on creating an S3 bucket and [here](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys)
for instructions to create AWS credentials.
See [here](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html) for instructions on creating an S3 bucket and
[here](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys) for instructions on creating AWS credentials.

#### Configuring Custom GCS Log Location
Create the GCP service account with read/write permission to the GCS log bucket.

1) Base64 encode the gcp json secret.
davinchia marked this conversation as resolved.
Show resolved Hide resolved
```
# The output of this command will be a Base64 string.
$ cat gcp.json | base64
```
2) Populate the gcs-log-creds secrets with the Base64-encoded credential. This is as simple as taking the encoded credential from the previous step
and adding it to the `secret-gcs-log-creds,yaml` file.
```
apiVersion: v1
kind: Secret
metadata:
name: gcs-log-creds
namespace: default
data:
gcp.json: <base64-encoded-string>
```

3) Replace the following variables in the `.env` file in the `kube/overlays/stable` directory:
```
# The GCS bucket to write logs in.
GCP_STORAGE_BUCKET=
# The path the GCS creds are written to. Unless you know what you are doing, use the below default value.
GOOGLE_APPLICATION_CREDENTIALS=/secrets/gcs-log-creds/gcp.json
```

See [here](https://cloud.google.com/storage/docs/creating-buckets) for instruction on creating a GCS bucket and
[here](https://cloud.google.com/iam/docs/creating-managing-service-account-keys#iam-service-account-keys-create-console) for instruction on creating GCP credentials.

### Launch Airbyte

Expand Down Expand Up @@ -197,10 +230,11 @@ kubectl exec -it airbyte-scheduler-6b5747df5c-bj4fx cat /tmp/workspace/8/0/logs.
```

### Persistent storage on GKE regional cluster
To manage persistent storage on GKE regional cluster you need to enable [CSI driver](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/gce-pd-csi-driver)\
When enabled you need to change storage class on [Volume config](../../kube/resources/volume-configs.yaml) and [Volume workspace](../../kube/resources/volume-workspace.yaml) \
Add `storageClassName: standard-rwo` in volume spec \
exemple for volume config:
Running Airbyte on GKE regional cluster requires enabling persistent regional storage. To do so, enable [CSI driver](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/gce-pd-csi-driver)
on GKE. After enabling, add `storageClassName: standard-rwo` to the [Config volume](../../kube/resources/volume-configs.yaml) and [Workspace volume](../../kube/resources/volume-workspace.yaml)
davinchia marked this conversation as resolved.
Show resolved Hide resolved
yamls.

`volume-configs.yaml` example:
```yaml
apiVersion: v1
kind: PersistentVolumeClaim
Expand Down