🎉 Destination Snowflake: Add option to stage encrypted files via S3 #12452

edgao · 2022-04-28T21:10:49Z

What

We want to be able to (a) write encrypted data to S3, and (b) COPY those encrypted files into Snowflake.

Also, fix destination-snowflake's purge_staging_data option (previously we were always deleting the output files)

How

Create a BlobDecorator abstract class, which represents modifications to how a single S3 (GCS, Azure Storage, etc.) file is uploaded. Implement an AesCbc subclass, which encrypts the file and sets additional metadata.
Modify BlobStorageOperations to accept 0 or more BlobDecorators. Make S3StorageOperations handle those decorators correctly. (GcsStorageOperations inherits that behavior)
Create an EncryptionConfig struct to represent different encryption strategies (currently: NoEncryption and AesCbcEnvelope)
Update destination-snowflake to accept an encryption entry inside its S3 Staging loading method, and pass that configuration down to SnowflakeS3StagingSqlOperations (which then injects the appropriate BlobDecorator into the S3StorageOperations, and adds the encryption key to the COPY command)

Some additional notes:

I couldn't find clear documentation on what the encryption format is supposed to look like; the implementation here was reverse-engineered from a Redshift UNLOAD command. I did run a local sync with encryption to Snowflake (see the blob on S3 and the AIRBYTE_DATABASE.AIRBYTE_SCHEMA._airbyte_raw_edgao_customers table in Snowflake)
New integration test config is added here. It uses ephemeral keys.
I didn't expose this feature in destination-s3, but it would be pretty easy. IMO it would be better to implement AES-GCM if we want to give users the option of encrypting their data though, since that's what the latest AWS SDK uses. (probably not difficult to implement)

🚨 User Impact 🚨

None. Existing configs will be use NoEncryption.

Pre-merge Checklist

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

edgao · 2022-04-28T22:40:17Z

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2242313134
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2242313134
Python tests coverage:

Name                                                                                                                            Stmts   Miss  Cover
---------------------------------------------------------------------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                                                                                          2      0   100%
normalization/transform_catalog/reserved_keywords.py                                                                               13      0   100%
normalization/transform_catalog/__init__.py                                                                                         2      0   100%
normalization/destination_type.py                                                                                                  13      0   100%
normalization/__init__.py                                                                                                           4      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/airbyte_protocol.py     124      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/__init__.py               1      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/__init__.py                      2      0   100%
normalization/transform_catalog/destination_name_transformer.py                                                                   155      8    95%
normalization/transform_config/transform.py                                                                                       159     31    81%
normalization/transform_catalog/table_name_registry.py                                                                            174     34    80%
normalization/transform_catalog/utils.py                                                                                           34      7    79%
normalization/transform_catalog/dbt_macro.py                                                                                       22      7    68%
normalization/transform_catalog/catalog_processor.py                                                                              147     80    46%
normalization/transform_catalog/transform.py                                                                                       61     38    38%
normalization/transform_catalog/stream_processor.py                                                                               534    345    35%
---------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                                            1447    550    62%

tuliren

Amazing!

...destination-s3/src/main/java/io/airbyte/integrations/destination/s3/S3StorageOperations.java

...t/java/io/airbyte/integrations/destination/s3/AesCbcEnvelopeEncryptionBlobDecoratorTest.java

tuliren · 2022-05-02T20:30:57Z

airbyte-integrations/connectors/destination-snowflake/src/main/resources/spec.json

+                        "title": "Key-encrypting key",
+                        "description": "The type of key-encrypting key to use",
+                        "type": "object",
+                        "oneOf": [


You should test the spec in storybook to see if it works. Previously our UI does not support multiple-level oneOf specs very well.

This can be replaced with a simple text field. When it is not empty, it is a user provided key. When it is empty, it is an ephemeral key.

I think nested oneOf work well now (tested in storybook, and also with a local airbyte instance). There's some weirdness with empty objects, but that seems like an actual bug which is also breaking e.g. the Internal Staging option - #12457

but will switch to the text field anyway, it would make the spec simpler to read + parse

airbyte-integrations/connectors/destination-snowflake/src/main/resources/spec.json

michel-tricot · 2022-05-02T22:02:37Z

...destination-s3/src/main/java/io/airbyte/integrations/destination/s3/S3StorageOperations.java

      dataStream.transferTo(outputStream);
      succeeded = true;
    } catch (final Exception e) {
      LOGGER.error("Failed to load data into storage {}", objectPath, e);
      throw new RuntimeException(e);
    } finally {
+      outputStream.close();


if close fails, it won't execute the uploadManager cleanups

edgao · 2022-05-03T16:26:06Z

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2265056010
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2265056010
Python tests coverage:

Name                                                                                                                            Stmts   Miss  Cover
---------------------------------------------------------------------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                                                                                          2      0   100%
normalization/transform_catalog/reserved_keywords.py                                                                               13      0   100%
normalization/transform_catalog/__init__.py                                                                                         2      0   100%
normalization/destination_type.py                                                                                                  13      0   100%
normalization/__init__.py                                                                                                           4      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/airbyte_protocol.py     124      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/__init__.py               1      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/__init__.py                      2      0   100%
normalization/transform_catalog/destination_name_transformer.py                                                                   155      8    95%
normalization/transform_config/transform.py                                                                                       159     31    81%
normalization/transform_catalog/table_name_registry.py                                                                            174     34    80%
normalization/transform_catalog/utils.py                                                                                           34      7    79%
normalization/transform_catalog/dbt_macro.py                                                                                       22      7    68%
normalization/transform_catalog/catalog_processor.py                                                                              147     80    46%
normalization/transform_catalog/transform.py                                                                                       61     38    38%
normalization/transform_catalog/stream_processor.py                                                                               534    345    35%
---------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                                            1447    550    62%

edgao · 2022-05-03T19:40:02Z

/publish connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2265971371
❌ Failed to publish connectors/destination-snowflake
❌ Couldn't auto-bump version for connectors/destination-snowflake

edgao · 2022-05-03T21:24:09Z

/publish connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2266439299
❌ Failed to publish connectors/destination-snowflake
❌ Couldn't auto-bump version for connectors/destination-snowflake

edgao · 2022-05-04T03:31:22Z

/publish connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2267582754
🚀 Successfully published connectors/destination-snowflake
🚀 Auto-bumped version for connectors/destination-snowflake
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2267582754

…fix purge_staging_data (#12452)

github-actions bot added the area/connectors Connector related issues label Apr 28, 2022

edgao mentioned this pull request Apr 28, 2022

Empty config objects inside oneOfs cannot be saved #12457

Closed

edgao added 15 commits April 28, 2022 16:37

add ability to set user metadata

7476ade

add BlobDecorator interface

3ce6cd0

implement encryption; better metadata handling

e0989b0

start writing encrypted COPY logic

335ee73

add tests

d0a4980

add encryption config; code cleanup

435dade

snowflake plugs into encryption?

3ab805d

update metadata

8cb7580

fix purge_staging_data

aa9b5fc

remove debug log

81cc4ca

switch BlobDecorator to abstract class

08886cb

switch BlobStorageOperations to abstract class

7d7706f

better metadata handling; implement for GCS

b09e82d

check for ephemeral+nopurge

ac47dda

add encryption acceptance test

9ae867b

edgao force-pushed the edgao/snowflake_encrypted_staging branch from 2b47a2b to 9ae867b Compare April 28, 2022 23:38

update doc

4338024

github-actions bot added the area/documentation Improvements or additions to documentation label Apr 28, 2022

edgao requested review from tuliren and subodh1810 April 28, 2022 23:48

edgao marked this pull request as ready for review April 28, 2022 23:48

reuse random generator

007ac8c

tuliren approved these changes May 2, 2022

View reviewed changes

edgao added 2 commits May 2, 2022 14:42

use constants

2ef1251

use StandardCharsets

8b9d808

michel-tricot reviewed May 2, 2022

View reviewed changes

switch back to try with resources

5a547d8

edgao added 2 commits May 2, 2022 17:06

suppress spotbugs warnings

03bc234

switch aes spec to just a string

4d98c0a

edgao added 3 commits May 3, 2022 09:35

code smells

035b6a5

Merge branch 'master' into edgao/snowflake_encrypted_staging

da3374e

version bumps

8aaf084

auto-bump connector version

fd09304

octavia-squidington-iii temporarily deployed to more-secrets May 4, 2022 04:25 Inactive

edgao merged commit 43470a2 into master May 4, 2022

edgao deleted the edgao/snowflake_encrypted_staging branch May 4, 2022 14:31

edgao mentioned this pull request May 4, 2022

Build Encryption logic for files being generated in destinations #10009

Closed

octavia-squidington-iii mentioned this pull request May 4, 2022

Bump Airbyte version from 0.36.8-alpha to 0.36.9-alpha #12585

Merged

suhomud pushed a commit that referenced this pull request May 23, 2022

🎉 Destination Snowflake: Add option to stage encrypted files via S3; …

1f60582

…fix purge_staging_data (#12452)

edgao mentioned this pull request Jun 5, 2024

Destination redshift: Remove encrypted staging option #39141

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Destination Snowflake: Add option to stage encrypted files via S3 #12452

🎉 Destination Snowflake: Add option to stage encrypted files via S3 #12452

edgao commented Apr 28, 2022 •

edited

Loading

edgao commented Apr 28, 2022 •

edited by github-actions bot

Loading

tuliren left a comment

tuliren May 2, 2022

tuliren May 2, 2022

edgao May 2, 2022

michel-tricot May 2, 2022

edgao May 3, 2022

edgao commented May 3, 2022 •

edited by github-actions bot

Loading

edgao commented May 3, 2022 •

edited by github-actions bot

Loading

edgao commented May 3, 2022 •

edited by github-actions bot

Loading

edgao commented May 4, 2022 •

edited by github-actions bot

Loading

🎉 Destination Snowflake: Add option to stage encrypted files via S3 #12452

🎉 Destination Snowflake: Add option to stage encrypted files via S3 #12452

Conversation

edgao commented Apr 28, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

edgao commented Apr 28, 2022 • edited by github-actions bot Loading

tuliren left a comment

Choose a reason for hiding this comment

tuliren May 2, 2022

Choose a reason for hiding this comment

tuliren May 2, 2022

Choose a reason for hiding this comment

edgao May 2, 2022

Choose a reason for hiding this comment

michel-tricot May 2, 2022

Choose a reason for hiding this comment

edgao May 3, 2022

Choose a reason for hiding this comment

edgao commented May 3, 2022 • edited by github-actions bot Loading

edgao commented May 3, 2022 • edited by github-actions bot Loading

edgao commented May 3, 2022 • edited by github-actions bot Loading

edgao commented May 4, 2022 • edited by github-actions bot Loading

edgao commented Apr 28, 2022 •

edited

Loading

edgao commented Apr 28, 2022 •

edited by github-actions bot

Loading

edgao commented May 3, 2022 •

edited by github-actions bot

Loading

edgao commented May 3, 2022 •

edited by github-actions bot

Loading

edgao commented May 3, 2022 •

edited by github-actions bot

Loading

edgao commented May 4, 2022 •

edited by github-actions bot

Loading