Fix Terraform module execution error (#600)

* added .gitignore * added adf terraform role to global.yml * Added DynamoDB permission to adf-codebuild-policy in deployment/global.yml * added adf-terraform-role to Organization Policy adf-build/global.yml * added organizations:ListChildren to OrganizationsReadOnlyPolicy * added DynamoDB table for terraform lock to adf-bootstrap/deployment/global.yml and regional.yml * added DynamoDB table for terraform lock to adf-bootstrap/deployment/global.yml * added sample-terraform repository * Added tfvars examples * clean comments * pull request template * Define PULL REQUEST * clean PULL_REQUEST_TEMPLATE.md * Update samples/sample-terraform/README.md Co-authored-by: Stewart Wallace <[email protected]> * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/get_accounts.py Co-authored-by: Stewart Wallace <[email protected]> * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/get_accounts.py clean comments Co-authored-by: Stewart Wallace <[email protected]> * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/get_accounts.py Co-authored-by: Stewart Wallace <[email protected]> * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/get_accounts.py clean comments Co-authored-by: Stewart Wallace <[email protected]> * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/get_accounts.py Co-authored-by: Stewart Wallace <[email protected]> * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/get_accounts.py Co-authored-by: Stewart Wallace <[email protected]> * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/get_accounts.py remove comments Co-authored-by: Stewart Wallace <[email protected]> * removed .gitignore from sample-terraform * Update samples/sample-terraform/README.md add details to documenation Co-authored-by: Simon <[email protected]> * renamed tf_sec to tf_scan in sample-terraform * added more details to README.md regarding REGIONS parameters * removed restart_execution_on_update: true * Update samples/sample-terraform/README.md Rename parameter from MASTER_ACCOUNT_ID to MANAGEMENT_ACCOUNT_ID Co-authored-by: Simon <[email protected]> * Update samples/sample-terraform/README.md Co-authored-by: Simon <[email protected]> * Update samples/sample-terraform/README.md Co-authored-by: Simon <[email protected]> * Update samples/sample-terraform/README.md Co-authored-by: Simon <[email protected]> * added latest terraform version in sample-terraform example * added terraform version support in documetation * fixed README.md format * added tf pipeline documentation in user guide * added s3 public access block to terraform sample * removed TF_STAGE as adf_terraform.sh input parameter * removed AWS_DEFAULT_REGION from tf_apply and tf_plan * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/adf_terraform.sh Co-authored-by: Simon <[email protected]> * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/get_accounts.py Co-authored-by: Simon <[email protected]> * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/get_accounts.py Co-authored-by: Simon <[email protected]> * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/get_accounts.py Co-authored-by: Simon <[email protected]> * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/get_accounts.py Co-authored-by: Simon <[email protected]> * revert to original PULL_REQUEST_TEMPLATE.md * clean .gitignore * add logging module * added main * undo linter changes * removed artifact from tf_apply * added ondemand capacity to DynamoDB table * removed terraform role. This could be added in global-iam by the user * added an example of terraform role. It must be uncommented by the user to enable tf extension * added tf pre-requisites to user-guide * global vars to upper case * added tfrun function to avoid code repetition and add functionality to copy tf plan to S3 bucket in the deployment account * clean documentation * Update docs/user-guide.md Co-authored-by: Simon Kok <[email protected]> * Update docs/user-guide.md Co-authored-by: Simon Kok <[email protected]> * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/get_accounts.py Co-authored-by: Simon Kok <[email protected]> * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/get_accounts.py Co-authored-by: Simon Kok <[email protected]> * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/get_accounts.py Co-authored-by: Simon Kok <[email protected]> * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/get_accounts.py Co-authored-by: Simon Kok <[email protected]> * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/get_accounts.py Co-authored-by: Simon Kok <[email protected]> * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/get_accounts.py Co-authored-by: Simon Kok <[email protected]> * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/get_accounts.py Co-authored-by: Simon Kok <[email protected]> * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/get_accounts.py Co-authored-by: Simon Kok <[email protected]> * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/get_accounts.py Co-authored-by: Simon Kok <[email protected]> * Update docs/user-guide.md Co-authored-by: Simon Kok <[email protected]> * Update docs/user-guide.md Co-authored-by: Simon Kok <[email protected]> * Update docs/user-guide.md Co-authored-by: Simon Kok <[email protected]> * Update docs/user-guide.md Co-authored-by: Simon Kok <[email protected]> * Update docs/user-guide.md Co-authored-by: Simon Kok <[email protected]> * Update docs/user-guide.md Co-authored-by: Simon Kok <[email protected]> * Update samples/sample-terraform/tf/s3.tf Co-authored-by: Simon Kok <[email protected]> * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/example-global-iam.yml Co-authored-by: Simon Kok <[email protected]> * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/adf_terraform.sh Co-authored-by: Simon Kok <[email protected]> * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/get_accounts.py Co-authored-by: Simon Kok <[email protected]> * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/adf_terraform.sh Co-authored-by: Simon Kok <[email protected]> * removed extra ] * changed session name * move TERRAFORM_VERSION variable to buildspec.yml * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/get_accounts.py Co-authored-by: Simon Kok <[email protected]> * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/get_accounts.py Co-authored-by: Simon Kok <[email protected]> * renamed environment variables * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/get_accounts.py Co-authored-by: Simon Kok <[email protected]> * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/get_accounts.py Co-authored-by: Simon Kok <[email protected]> * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/adf_terraform.sh Co-authored-by: Simon Kok <[email protected]> * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/adf_terraform.sh Co-authored-by: Simon Kok <[email protected]> * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/adf_terraform.sh Co-authored-by: Simon Kok <[email protected]> * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/adf_terraform.sh Co-authored-by: Simon Kok <[email protected]> * Update docs/user-guide.md Co-authored-by: Simon Kok <[email protected]> * Update docs/user-guide.md Co-authored-by: Simon Kok <[email protected]> * fixed toc identation * added details to terraform sections * Update docs/user-guide.md Co-authored-by: Simon Kok <[email protected]> * Update docs/user-guide.md Co-authored-by: Simon Kok <[email protected]> * Update docs/user-guide.md Co-authored-by: Simon Kok <[email protected]> * Update docs/user-guide.md Co-authored-by: Simon Kok <[email protected]> * Update docs/user-guide.md Co-authored-by: Simon Kok <[email protected]> * Update samples/sample-terraform/README.md Co-authored-by: Simon Kok <[email protected]> * Update samples/sample-terraform/README.md Co-authored-by: Simon Kok <[email protected]> * renamed terraform to upper case and align content of README.md file in sample-terraform * Update samples/sample-terraform/README.md Co-authored-by: Simon Kok <[email protected]> * Update samples/sample-terraform/buildspec.yml Co-authored-by: Simon Kok <[email protected]> * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/adf_terraform.sh Co-authored-by: Simon Kok <[email protected]> * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/adf_terraform.sh Co-authored-by: Simon Kok <[email protected]> * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/adf_terraform.sh Co-authored-by: Simon Kok <[email protected]> * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/adf_terraform.sh Co-authored-by: Simon Kok <[email protected]> * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/adf_terraform.sh Co-authored-by: Simon Kok <[email protected]> * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/adf_terraform.sh Co-authored-by: Simon Kok <[email protected]> * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/adf_terraform.sh Co-authored-by: Simon Kok <[email protected]> * removed Terraform IAM role and policy * added support to partitions * Update docs/user-guide.md Co-authored-by: Simon Kok <[email protected]> * Update docs/user-guide.md Co-authored-by: Simon Kok <[email protected]> * Update docs/user-guide.md Co-authored-by: Simon Kok <[email protected]> * Update docs/user-guide.md Co-authored-by: Simon Kok <[email protected]> * Update docs/user-guide.md Co-authored-by: Simon Kok <[email protected]> * Update samples/sample-terraform/README.md Co-authored-by: Simon Kok <[email protected]> * Update samples/sample-terraform/README.md Co-authored-by: Simon Kok <[email protected]> * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/example-global-iam.yml Co-authored-by: Simon Kok <[email protected]> * removed TerraformLockTable resource as already defined in regional.yml * Update docs/user-guide.md Co-authored-by: Simon Kok <[email protected]> * Update docs/user-guide.md Co-authored-by: Simon Kok <[email protected]> * Update docs/user-guide.md Co-authored-by: Simon Kok <[email protected]> * Update samples/sample-terraform/README.md Co-authored-by: Simon Kok <[email protected]> * Update samples/sample-terraform/README.md Co-authored-by: Simon Kok <[email protected]> * added newline character * added newline character * added newline character * added docstring * read CROSS_ACCOUNT_ACCESS_ROLE from parameter store * replaced sample adf terraform policy example * replaced sample adf terraform policy example * added details to adf terraform role description * removed trailing spaces * added aws partition variable * added init stage only as option * import extensions parameter in SSM Parameter Store of Management account * import extensions parameter in SSM Parameter Store of Deployment account (all regions) * added condition on DynamoDB table. Deploy only if Tf extension is enabled * added example in adfconfig related to Tf extension * added details related to terraform extension * added details related to terraform extension * fixed indentation * renamed paginator variable * Resolve W1514 - use open with encoding * Fix user guide target via tags * changed default codebuild image to STANDARD_5_0 * Fix YAML lint issues * Fix CFN lint issues * Fix MegaLint issues * Adding in default values for extensions * Remove redundant paginator This is already available at: src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/paginator.py * Fix TF doc comments and links * Fix TF line length findings * Revert default CodeBuild container, would introduce a breaking change The default CodeBuild container image to use cannot be changed without introducing a breaking change. A breaking change would imply a major version release. Since we are adding TF support in v3.2.0, a minor version release, we cannot modify this yet. It is on the roadmap though for the next major release :). * Replace redundant code writing extension parameters * fixed bash syntax * fixed parameter type to string * merge to main repo * merge user_guide * merge to master * merge to master2 * fixed double quotes bash syntax * added newline * fixed bash syntax * added tfstate-lockfile DynamoDB table * fixed ADFTerraformExtensionEnabled condition * removed trailing whitespace * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/adf_terraform.sh Co-authored-by: Simon Kok <[email protected]> * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/adf_terraform.sh Co-authored-by: Simon Kok <[email protected]> * Update get_accounts.py * Update src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/get_accounts.py * added terraform destroy capabilities to terraform script * added terraform destroy buildspec file to samples * added terraform destroy capability to user documentation * removed trailing whitespace * removed syntax error * Fix linting / style issues --------- Co-authored-by: Stefano Montanelli <[email protected]> Co-authored-by: Stewart Wallace <[email protected]> Co-authored-by: Simon <[email protected]> Co-authored-by: Stewart Wallace <[email protected]> Co-authored-by: Simon Kok <[email protected]>
awslabs · Jun 1, 2023 · 71c400b · 71c400b
1 parent b32326c
commit 71c400b
Show file tree

Hide file tree

Showing 7 changed files with 70 additions and 9 deletions.
diff --git a/docs/user-guide.md b/docs/user-guide.md
@@ -1077,6 +1077,8 @@ stages defined in the following CodeBuild build specification:
   run a Terraform plan.
 - `tf_apply.yml`: get the list of accounts from the organization and
   run a Terraform plan and apply.
+- `tf_destroy.yml`: get the list of accounts from the organization and
+  run a Terraform plan and destroy.
 
 An optional approval step could be added between plan and apply as
 shown in the pipeline definition below.
@@ -1132,6 +1134,9 @@ pipelines:
       - name: terraform-apply
         properties:
           spec_filename: tf_apply.yml  # Terraform apply
+      - name: terraform-destroy # (optional stage)
+        properties:
+          spec_filename: tf_destroy.yml  # Terraform destroy
 ```
 
 1. Add a sample-terraform pipeline in ADF `deployment-map.yml` as shown above.

diff --git a/linters/custom-adf-dict.txt b/linters/custom-adf-dict.txt
@@ -48,9 +48,11 @@ sdkman
 stefanzweifel
 stubber
 tfapply
+tfdestroy
 tfinit
 tflint
 tflocktable
+tfplandestroy
 tfrun
 tfstate
 tfvars

diff --git a/samples/sample-terraform/tf_destroy.yml b/samples/sample-terraform/tf_destroy.yml
@@ -0,0 +1,18 @@
+version: 0.2
+
+env:
+  variables:
+    TF_VAR_TARGET_ACCOUNT_ROLE: adf-terraform-role  # The IAM Role Terraform will assume to deploy resources
+    TF_IN_AUTOMATION: true
+    TF_STAGE: "destroy"
+    TF_CLI_ARGS: "-no-color"
+
+phases:
+  install:
+    runtime-versions:
+      python: 3.9
+
+  build:
+    commands:
+      - python adf-build/helpers/terraform/get_accounts.py
+      - bash adf-build/helpers/terraform/adf_terraform.sh
diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/global.yml b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/global.yml
@@ -64,6 +64,14 @@ Parameters:
     Type: String
     Default: "adf-"
 
+  ADFTerraformExtension:
+    Type: "AWS::SSM::Parameter::Value<String>"
+    Default: /adf/extensions/terraform/enabled
+
+Conditions:
+  ADFTerraformExtensionEnabled:
+    !Equals [!Ref ADFTerraformExtension, "True"]
+
 Globals:
   Function:
     CodeUri: lambda_codebase
@@ -1457,6 +1465,21 @@ Resources:
           RoleArn: !GetAtt PipelineCloudWatchEventRole.Arn
           Id: adf-codepipeline-trigger-pipeline
 
+  TerraformLockTable:
+    Condition: ADFTerraformExtensionEnabled
+    Type: "AWS::DynamoDB::Table"
+    DeletionPolicy: Retain
+    UpdateReplacePolicy: Retain
+    Properties:
+      AttributeDefinitions:
+        - AttributeName: LockID
+          AttributeType: S
+      KeySchema:
+        - AttributeName: LockID
+          KeyType: HASH
+      BillingMode: PAY_PER_REQUEST
+      TableName: adf-tflocktable
+
 Outputs:
   ADFVersionNumber:
     Value: !Ref ADFVersion

diff --git a/...lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/regional.yml b/...lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/regional.yml
@@ -15,9 +15,7 @@ Parameters:
 
 Conditions:
   ADFTerraformExtensionEnabled:
-    Fn::Equals:
-      - !Ref ADFTerraformExtension
-      - true
+    !Equals [!Ref ADFTerraformExtension, "True"]
 
 Resources:
   DeploymentFrameworkRegionalS3Bucket:

diff --git a/...e/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/adf_terraform.sh b/...e/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/adf_terraform.sh
@@ -10,12 +10,14 @@ tfinit(){
     S3_BUCKET_REGION_NAME=$(aws ssm get-parameter --name "/cross_region/s3_regional_bucket/$AWS_REGION" --region "$AWS_DEFAULT_REGION" | jq .Parameter.Value | sed s/\"//g)
     mkdir -p "${CURRENT}/tmp/${TF_VAR_TARGET_ACCOUNT_ID}-${AWS_REGION}"
     cd "${CURRENT}/tmp/${TF_VAR_TARGET_ACCOUNT_ID}-${AWS_REGION}" || exit
-    cp -R "$CURRENT/tf/*" "${CURRENT}/tmp/${TF_VAR_TARGET_ACCOUNT_ID}-${AWS_REGION}"
+    cp -R "${CURRENT}"/tf/* "${CURRENT}/tmp/${TF_VAR_TARGET_ACCOUNT_ID}-${AWS_REGION}"
     # if account related variables exist copy the folder in the work directory
-    if [ -d "$CURRENT/tfvars/$TF_VAR_TARGET_ACCOUNT_ID" ]; then
-        cp -R "${CURRENT}/tfvars/${TF_VAR_TARGET_ACCOUNT_ID}/*" "${CURRENT}/tmp/${TF_VAR_TARGET_ACCOUNT_ID}-${AWS_REGION}"
+    if [ -d "${CURRENT}/tfvars/${TF_VAR_TARGET_ACCOUNT_ID}" ]; then
+        cp -R "${CURRENT}/tfvars/${TF_VAR_TARGET_ACCOUNT_ID}"/* "${CURRENT}/tmp/${TF_VAR_TARGET_ACCOUNT_ID}-${AWS_REGION}"
+    fi
+    if [ -f "${CURRENT}/tfvars/global.auto.tfvars" ]; then
+        cp -R "${CURRENT}/tfvars/global.auto.tfvars" "${CURRENT}/tmp/${TF_VAR_TARGET_ACCOUNT_ID}-${AWS_REGION}"
     fi
-    cp -R "${CURRENT}/tfvars/global.auto.tfvars" "${CURRENT}/tmp/${TF_VAR_TARGET_ACCOUNT_ID}-${AWS_REGION}"
     terraform init \
         -backend-config "bucket=$S3_BUCKET_REGION_NAME" \
         -backend-config "region=$AWS_REGION" \
@@ -30,7 +32,7 @@ tfinit(){
 tfplan(){
     DATE=$(date +%Y-%m-%d)
     TS=$(date +%Y%m%d%H%M%S)
-    bash "$CURRENT/adf-build/helpers/sts.sh" "$TF_VAR_TARGET_ACCOUNT_ID" "$TF_VAR_TARGET_ACCOUNT_ROLE"
+    bash "${CURRENT}/adf-build/helpers/sts.sh" "${TF_VAR_TARGET_ACCOUNT_ID}" "${TF_VAR_TARGET_ACCOUNT_ROLE}"
     terraform plan -out "${ADF_PROJECT_NAME}-${TF_VAR_TARGET_ACCOUNT_ID}" 2>&1 | tee -a "${ADF_PROJECT_NAME}-${TF_VAR_TARGET_ACCOUNT_ID}-${TS}.log"
     # Save Terraform plan results to the S3 bucket
     aws s3 cp "${ADF_PROJECT_NAME}-${TF_VAR_TARGET_ACCOUNT_ID}-${TS}.log" "s3://${S3_BUCKET_REGION_NAME}/${ADF_PROJECT_NAME}/tf-plan/${DATE}/${TF_VAR_TARGET_ACCOUNT_ID}/${ADF_PROJECT_NAME}-${TF_VAR_TARGET_ACCOUNT_ID}-${TS}.log"
@@ -39,6 +41,12 @@ tfplan(){
 tfapply(){
     terraform apply "${ADF_PROJECT_NAME}-${TF_VAR_TARGET_ACCOUNT_ID}"
 }
+tfplandestroy(){
+    terraform plan -destroy -out "${ADF_PROJECT_NAME}-${TF_VAR_TARGET_ACCOUNT_ID}-destroy"
+}
+tfdestroy(){
+    terraform apply "${ADF_PROJECT_NAME}-${TF_VAR_TARGET_ACCOUNT_ID}-destroy"
+}
 tfrun(){
     export TF_VAR_TARGET_ACCOUNT_ID=$ACCOUNT_ID
     echo "Running terraform $TF_STAGE on account $ACCOUNT_ID and region $REGION"
@@ -60,6 +68,13 @@ tfrun(){
         tfplan
         tfapply
         set +e
+    elif [[ "$TF_STAGE" = "destroy" ]]
+    then
+        set -e
+        tfinit
+        tfplandestroy
+        tfdestroy
+        set +e
     else
         echo "Invalid Terraform stage: TF_STAGE = $TF_STAGE"
         exit 1

diff --git a/...se/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/get_accounts.py b/...se/initial_commit/bootstrap_repository/adf-build/shared/helpers/terraform/get_accounts.py
@@ -96,7 +96,7 @@ def get_accounts_from_ous():
         'organizations',
         (
             f'arn:{PARTITION}:sts::{MANAGEMENT_ACCOUNT_ID}:role/'
-            f'{CROSS_ACCOUNT_ACCESS_ROLE}-readonly',
+            f'{CROSS_ACCOUNT_ACCESS_ROLE}-readonly'
         ),
         'getRootAccountIDs',
     )