rapid-snowflake-deployment

This release uses Terraform 13 to rapidly deploy Snowflake resources to support automated data ingestion and exports to AWS S3.

Updates:

• Code to create an AWS CI user & role (which the user can adopt) to support automated deployments using CI/CD.
• Automated Snowflake deployment pipelines with GitHub actions CI/CD workflows.
• Snowflake tasks data export module to automate data unloading to S3.

The tooling can create the following resources and their dependencies:

• AWS S3 state bucket & DynamoDB lock table
• AWS S3 data lake
• AWS IAM roles for PUT & GET on S3 files (necessary for imports to Snowflake using Snowpipes and exports to S3 using External Stages)
• Snowpipes
• External Stages
• Storage Integrations

Rapid Snowflake Deployment

This initial release uses terraform to rapidly deploy Snowflake resources for data ingestion from AWS S3.

It uses terraform workspaces to separate environments and simplify parameter declarations.

The tooling can create the following resources and their dependencies:

AWS S3 state bucket & DynamoDB lock table
AWS S3 data lake
AWS IAM roles for PUT & GET on S3 files (necessary for imports to Snowflake using Snowpipes and exports to S3 using External Stages)
Snowpipes
External Stages
Storage Integrations

Additions from v0.1.0 is the removal of a circular dependency during initial set up of the DynamoDB lock table.

Rapid Snowflake Deployment

This initial release uses terraform to rapidly deploy Snowflake resources for data ingestion from AWS S3.

It uses terraform workspaces to separate environments and simplify parameter declarations.

The tooling can create the following resources and their dependencies:

• AWS S3 state bucket & DynamoDB lock table
• AWS S3 data lake
• AWS IAM roles for PUT & GET on S3 files (necessary for imports to Snowflake using Snowpipes and exports to S3 using External Stages)
• Snowpipes
• External Stages
• Storage Integrations