From e28a62bc6a965741aa9baeecda36e187a29f163a Mon Sep 17 00:00:00 2001 From: Edward Gao Date: Fri, 21 Oct 2022 11:43:09 -0700 Subject: [PATCH] require google-managed encryption key --- docs/integrations/destinations/bigquery.md | 2 ++ docs/integrations/destinations/gcs.md | 2 ++ 2 files changed, 4 insertions(+) diff --git a/docs/integrations/destinations/bigquery.md b/docs/integrations/destinations/bigquery.md index eb9ba2c9157f..994adcce3001 100644 --- a/docs/integrations/destinations/bigquery.md +++ b/docs/integrations/destinations/bigquery.md @@ -36,6 +36,8 @@ To use a Google Cloud Storage bucket: 3. Grant the [`Storage Object Admin` role](https://cloud.google.com/storage/docs/access-control/iam-roles#standard-roles) to the Google Cloud [Service Account](https://cloud.google.com/iam/docs/service-accounts). 4. Make sure your Cloud Storage bucket is accessible from the machine running Airbyte. The easiest way to verify if Airbyte is able to connect to your bucket is via the check connection tool in the UI. +Your bucket must be encrypted using a Google-managed encryption key (this is the default setting when creating a new bucket). We currently do not support buckets using customer-managed encryption keys (CMEK). You can view this setting under the "Configuration" tab of your GCS bucket, in the `Encryption type` row. + #### Using `INSERT` You can use BigQuery's [`INSERT`](https://cloud.google.com/bigquery/docs/reference/standard-sql/dml-syntax) statement to upload data directly from your source to BigQuery. While this is faster to set up initially, we strongly recommend not using this option for anything other than a quick demo. Due to the Google BigQuery SDK client limitations, using `INSERT` is 10x slower than using a Google Cloud Storage bucket, and you may see some failures for big datasets and slow sources (For example, if reading from a source takes more than 10-12 hours). For more details, refer to https://github.com/airbytehq/airbyte/issues/3549 diff --git a/docs/integrations/destinations/gcs.md b/docs/integrations/destinations/gcs.md index 2e1661fa0e5e..aa07d098ed17 100644 --- a/docs/integrations/destinations/gcs.md +++ b/docs/integrations/destinations/gcs.md @@ -31,6 +31,8 @@ The Airbyte GCS destination allows you to sync data to cloud storage buckets. Ea Currently, only the [HMAC key](https://cloud.google.com/storage/docs/authentication/hmackeys) is supported. More credential types will be added in the future, please [submit an issue](https://github.com/airbytehq/airbyte/issues/new?assignees=&labels=type%2Fenhancement%2C+needs-triage&template=feature-request.md&title=) with your request. +Additionally, your bucket must be encrypted using a Google-managed encryption key (this is the default setting when creating a new bucket). We currently do not support buckets using customer-managed encryption keys (CMEK). You can view this setting under the "Configuration" tab of your GCS bucket, in the `Encryption type` row. + ⚠️ Please note that under "Full Refresh Sync" mode, data in the configured bucket and path will be wiped out before each sync. We recommend you to provision a dedicated S3 resource for this sync to prevent unexpected data deletion from misconfiguration. ⚠️ The full path of the output data is: