Upgrade CDK (v2.119), SAM (v1.107), Jinja2 (v3.1.3), and others to la…

…test compatible version (#676)

* Upgrade CDK (v2.113), SAM (v1.103), and others to latest compatible version

**Why?**

To use the latest versions as made available in the meanwhile.

**What?**

* Upgrade CDK (v2.113), SAM (v1.103), and others to latest compatible version
* Remove Python based awscli v1, rely on default AWS CLI part of CodeBuild
  instead.
* Fix Terraform Enabled Parameter Store test
* Hide logs emitted by tests.

* Upgrade CDK (v2.115), SAM (v1.105), Python (v3.12) and others to latest compatible

**Why?**

To use the latest versions as made available in the meanwhile.

**What?**

* Upgrade CDK (v2.115), SAM (v1.105), and others to latest compatible version
* Update to Python v3.12.
* Ensure Regular Expressions use raw strings where required.

* Upgrade CDK (v2.118.0) and ARM64

**Why?**

To use the latest versions as made available in the meanwhile.

**What?**

* Upgrade CDK (v2.118.0).
* Upgrade AWS SAM CLI to v1.107.0.
* Switch to the ARM64 architecture for Lambda functions.
* Add the new required ImageId property for Cloud9 instances in the service
  catalog sample.

* Undo ARM64 for all Lambda functions

Reverting, due to `ruamel-yaml-clib==0.2.8(wheel)` dependency error.

* Bump jinja2

Bumps [jinja2](https://github.com/pallets/jinja) from 3.1.2 to 3.1.3.
- [Release notes](https://github.com/pallets/jinja/releases)
- [Changelog](https://github.com/pallets/jinja/blob/main/CHANGES.rst)
- [Commits](pallets/jinja@3.1.2...3.1.3)

---
updated-dependencies:
- dependency-name: jinja2
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>

* Bump jinja2 from 3.1.2 to 3.1.3 in /src/lambda_codebase/initial_commit

Bumps [jinja2](https://github.com/pallets/jinja) from 3.1.2 to 3.1.3.
- [Release notes](https://github.com/pallets/jinja/releases)
- [Changelog](https://github.com/pallets/jinja/blob/main/CHANGES.rst)
- [Commits](pallets/jinja@3.1.2...3.1.3)

---
updated-dependencies:
- dependency-name: jinja2
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>

* Upgrade ADF Workflow to Python 3.12

* Fix repeated headers blocker in providers guide

* Upgrade to CDK v2.119

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

.github/workflows/adf.yml

@@ -7,7 +7,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ["3.9"]
+        python-version: ["3.12"]
 
     steps:
       - name: Checkout Repo

Makefile

@@ -17,7 +17,7 @@ test:
 
 lint:
 	# Linter performs static analysis to catch latent bugs
-	find src -iname "*.py" -not -path "src/.aws-sam/*" | xargs pylint --rcfile .pylintrc
+	find src -iname "*.py" -not -path "src/.aws-sam/*" | xargs pylint --verbose --rcfile .pylintrc
 	find src -iname "*.yml" -o -iname "*.yaml" -not -path "src/.aws-sam/*" | xargs yamllint -c .yamllint.yml
 	cfn-lint
 

docs/providers-guide.md

@@ -1,5 +1,8 @@
 # Providers Guide
 
+<!-- markdownlint-disable MD024 -->
+<!-- ^ Allow repeated headers to be used in this file -->
+
 Provider types and their properties can be defined as default config for a
 pipeline. But also at the stage level of a pipeline to structure the source,
 build, test, approval, deploy or invoke actions.

docs/user-guide.md

@@ -954,8 +954,8 @@ version: 0.2
 phases:
   install:
     runtime-versions:
-      python: 3.11
-      nodejs: 18
+      python: 3.12
+      nodejs: 20
   pre_build:
     commands:
       - aws s3 cp s3://$S3_BUCKET_NAME/adf-build/ adf-build/ --recursive --quiet

requirements-dev.txt

@@ -1,8 +1,8 @@
-cfn-lint==0.78.2
-isort==5.12.0
+cfn-lint==0.83.8
+isort==5.13.2
 mock==5.1.0
-pylint==2.17.4
-pytest~=7.4.0
-pytest-cov==3.0.0
-tox==3.28.0
-yamllint==1.32.0
+pylint==3.0.3
+pytest~=7.4.4
+pytest-cov==4.1.0
+tox==4.11.4
+yamllint==1.33.0

requirements.txt

@@ -1,5 +1,6 @@
-astroid==2.15.6
-boto3==1.28.8
-botocore==1.31.8
+astroid==3.0.2
+aws-sam-cli==1.107.0
+boto3==1.34.17
+botocore==1.34.17
 pyyaml~=6.0.1
 schema==0.7.5

samples/sample-cdk-app/buildspec.yml

@@ -3,8 +3,8 @@ version: 0.2
 phases:
   install:
     runtime-versions:
-      python: 3.11
-      nodejs: 18
+      python: 3.12
+      nodejs: 20
     commands:
       - aws s3 cp s3://$S3_BUCKET_NAME/adf-build/ adf-build/ --recursive --quiet
       - pip install -r adf-build/requirements.txt -q

samples/sample-codebuild-vpc/buildspec.yml

@@ -3,7 +3,7 @@ version: 0.2
 phases:
   install:
     runtime-versions:
-      python: 3.11
+      python: 3.12
     commands:
       # It will connect through the VPC to fetch all the resources.
       # Make sure the subnets and security groups are configured such that

samples/sample-codebuild-vpc/testspec.yml

@@ -3,7 +3,7 @@ version: 0.2
 phases:
   install:
     runtime-versions:
-      python: 3.11
+      python: 3.12
 
   build:
     commands:

samples/sample-ec2-with-codedeploy/buildspec.yml

@@ -3,7 +3,7 @@ version: 0.2
 phases:
   install:
     runtime-versions:
-      python: 3.11
+      python: 3.12
     commands:
       - aws s3 cp s3://$S3_BUCKET_NAME/adf-build/ adf-build/ --recursive --quiet
       - pip install -r adf-build/requirements.txt -q

samples/sample-ecr-repository/buildspec.yml

@@ -3,7 +3,7 @@ version: 0.2
 phases:
   install:
     runtime-versions:
-      python: 3.11
+      python: 3.12
     commands:
       - aws s3 cp s3://$S3_BUCKET_NAME/adf-build/ adf-build/ --recursive --quiet
       - pip install -r adf-build/requirements.txt -q

samples/sample-ecs-cluster/buildspec.yml

@@ -3,7 +3,7 @@ version: 0.2
 phases:
   install:
     runtime-versions:
-      python: 3.11
+      python: 3.12
     commands:
       - aws s3 cp s3://$S3_BUCKET_NAME/adf-build/ adf-build/ --recursive --quiet
       - pip install -r adf-build/requirements.txt -q

samples/sample-expunge-vpc/buildspec.yml

@@ -3,7 +3,7 @@ version: 0.2
 phases:
   install:
     runtime-versions:
-      python: 3.11
+      python: 3.12
     commands:
       - aws s3 cp s3://$S3_BUCKET_NAME/adf-build/ adf-build/ --recursive --quiet
       - pip install -r adf-build/requirements.txt -q

samples/sample-expunge-vpc/template.yml

@@ -34,7 +34,7 @@ Resources:
       Handler: lambda_function.lambda_handler
       MemorySize: 128
       Role: !GetAtt LambdaVPCPolicyRole.Arn
-      Runtime: python3.10
+      Runtime: python3.12
       Timeout: 600
       Environment:
         Variables:

samples/sample-fargate-node-app/buildspec.yml

@@ -3,8 +3,7 @@ version: 0.2
 phases:
   install:
     runtime-versions:
-      docker: 20
-      python: 3.11
+      python: 3.12
 
   build:
     commands:

samples/sample-iam/buildspec.yml

@@ -3,7 +3,7 @@ version: 0.2
 phases:
   install:
     runtime-versions:
-      python: 3.11
+      python: 3.12
     commands:
       - aws s3 cp s3://$S3_BUCKET_NAME/adf-build/ adf-build/ --recursive --quiet
       - pip install -r adf-build/requirements.txt -q

samples/sample-mono-repo/apps/alpha/buildspec.yml

@@ -7,7 +7,7 @@ env:
 phases:
   install:
     runtime-versions:
-      python: 3.11
+      python: 3.12
     commands:
       - cd $INFRASTRUCTURE_ROOT_DIR
       - aws s3 cp s3://$S3_BUCKET_NAME/adf-build/ adf-build/ --recursive --quiet

samples/sample-mono-repo/apps/beta/buildspec.yml

@@ -7,7 +7,7 @@ env:
 phases:
   install:
     runtime-versions:
-      python: 3.11
+      python: 3.12
     commands:
       - cd $INFRASTRUCTURE_ROOT_DIR
       - aws s3 cp s3://$S3_BUCKET_NAME/adf-build/ adf-build/ --recursive --quiet

samples/sample-rdk-rules/buildspec.yml

@@ -2,8 +2,8 @@ version: 0.2
 phases:
   install:
     runtime-versions:
-      python: 3.11
-      nodejs: 18
+      python: 3.12
+      nodejs: 20
     commands:
       - aws s3 cp s3://$S3_BUCKET_NAME/adf-build/ adf-build/ --recursive --quiet
       - pip install -r adf-build/requirements.txt -q

samples/sample-rdk-rules/config-rules/EC2_CHECKS_TERMINATION_PROTECTION_ADF/parameters.json

@@ -3,7 +3,7 @@
     "Parameters": {
         "RuleName": "EC2_CHECKS_TERMINATION_PROTECTION_ADF",
         "Description": "EC2_CHECKS_TERMINATION_PROTECTION_ADF",
-        "SourceRuntime": "python3.10",
+        "SourceRuntime": "python3.12",
         "CodeKey": "EC2_CHECKS_TERMINATION_PROTECTION_ADFeu-central-1.zip",
         "InputParameters": "{}",
         "OptionalParameters": "{}",

samples/sample-rdk-rules/requirements.txt

@@ -1,3 +1,3 @@
 s3==3.0.0
-boto3==1.28.8
+boto3==1.34.17
 argparse==1.4.0

samples/sample-serverless-app/buildspec.yml

@@ -3,7 +3,7 @@ version: 0.2
 phases:
   install:
     runtime-versions:
-      python: 3.11
+      python: 3.12
 
   build:
     commands:

samples/sample-serverless-app/template.yml

@@ -17,7 +17,7 @@ Resources:
     Type: 'AWS::Serverless::Function'
     Properties:
       Handler: lambda_function.handler
-      Runtime: python3.10
+      Runtime: python3.12
       CodeUri: .
       Description: Sample Lambda Function
       MemorySize: 128

samples/sample-service-catalog-product/buildspec.yml

@@ -3,7 +3,7 @@ version: 0.2
 phases:
   install:
     runtime-versions:
-      python: 3.11
+      python: 3.12
     commands:
       - aws s3 cp s3://$S3_BUCKET_NAME/adf-build/ adf-build/ --recursive --quiet
       - pip install -r adf-build/requirements.txt -q

samples/sample-service-catalog-product/productX/template.yml

@@ -50,6 +50,7 @@ Resources:
       AutomaticStopTimeMinutes: !Ref AutomaticStopTimeInMinutes
       Description: !Ref InstanceDescription
       InstanceType: !Ref InstanceType
+      ImageId: 'amazonlinux-2023-x86_64'
       Name: !Ref InstanceName
       OwnerArn: !Sub "arn:${AWS::Partition}:iam::${AWS::AccountId}:user/${UserName}"  # In this sample case 'sample-developer' from the IAM stack can be used here
       SubnetId:

samples/sample-terraform/tf_apply.yml

@@ -10,7 +10,7 @@ env:
 phases:
   install:
     runtime-versions:
-      python: 3.11
+      python: 3.12
 
   build:
     commands:

samples/sample-terraform/tf_destroy.yml

@@ -10,7 +10,7 @@ env:
 phases:
   install:
     runtime-versions:
-      python: 3.9
+      python: 3.12
 
   build:
     commands:

samples/sample-terraform/tf_plan.yml

@@ -10,7 +10,7 @@ env:
 phases:
   install:
     runtime-versions:
-      python: 3.11
+      python: 3.12
 
   build:
     commands:

samples/sample-vpc/buildspec.yml

@@ -3,7 +3,7 @@ version: 0.2
 phases:
   install:
     runtime-versions:
-      python: 3.11
+      python: 3.12
     commands:
       - aws s3 cp s3://$S3_BUCKET_NAME/adf-build/ adf-build/ --recursive --quiet
       - pip install -r adf-build/requirements.txt -q

src/lambda_codebase/account/main.py

@@ -5,6 +5,7 @@
 deployment account if required.
 """
 
+import os
 from typing import Mapping, Any, Tuple
 from dataclasses import dataclass, asdict
 import logging
@@ -28,7 +29,8 @@
 ORGANIZATION_CLIENT = boto3.client("organizations")
 SSM_CLIENT = boto3.client("ssm")
 LOGGER = logging.getLogger(__name__)
-LOGGER.setLevel(logging.INFO)
+LOGGER.setLevel(os.environ.get("ADF_LOG_LEVEL", logging.INFO))
+logging.basicConfig(level=logging.INFO)
 MAX_RETRIES = 120  # => 120 retries * 5 seconds = 10 minutes
 
 

src/lambda_codebase/account_processing/process_account_files.py

@@ -142,7 +142,7 @@ def process_account_list(all_accounts, accounts_in_file):
 
 
 def sanitize_account_name_for_snf(account_name):
-    return re.sub("[^a-zA-Z0-9_]", "_", account_name[:30])
+    return re.sub(r"[^a-zA-Z0-9_]", "_", account_name[:30])
 
 
 def start_executions(

src/lambda_codebase/account_processing/requirements.txt

@@ -1,3 +1,2 @@
-aws-xray-sdk==2.12.0
+aws-xray-sdk==2.12.1
 pyyaml~=6.0.1
-wrapt==1.14.1 # https://github.com/aws/aws-lambda-builders/issues/302

src/lambda_codebase/cross_region_bucket/main.py

@@ -7,12 +7,14 @@
 """
 
 
+import os
 from typing import Mapping, Any, Tuple, MutableMapping
 from dataclasses import dataclass, asdict
 import logging
 import json
 import secrets
-import string # pylint: disable=deprecated-module # https://www.logilab.org/ticket/2481
+import string  # pylint: disable=deprecated-module
+# ^ https://www.logilab.org/ticket/2481
 import boto3
 from cfn_custom_resource import ( # pylint: disable=unused-import
     lambda_handler,
@@ -34,7 +36,8 @@
 
 # Globals:
 LOGGER = logging.getLogger(__name__)
-LOGGER.setLevel(logging.INFO)
+LOGGER.setLevel(os.environ.get("ADF_LOG_LEVEL", logging.INFO))
+logging.basicConfig(level=logging.INFO)
 S3CLIENTS: MutableMapping[Region, S3Client] = {}
 SSM_CLIENT = boto3.client("ssm")
 

src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/global.yml

@@ -38,9 +38,7 @@ Parameters:
   ComputeType:
     Description: The Compute Type to use for AWS CodeBuild
     Type: String
-    # BUILD_GENERAL1_LARGE - For threading with large amounts of pipelines
-    # this is the most effective default:
-    Default: "BUILD_GENERAL1_LARGE"
+    Default: "BUILD_GENERAL1_SMALL"
     AllowedValues:
       - "BUILD_GENERAL1_SMALL"  # 3 GB memory, 2 vCPU
       - "BUILD_GENERAL1_MEDIUM"  # 7 GB memory, 4 vCPU
@@ -75,15 +73,15 @@ Conditions:
 Globals:
   Function:
     CodeUri: lambda_codebase
-    Runtime: python3.10
+    Runtime: python3.12
 
 Resources:
   LambdaLayerVersion:
     Type: "AWS::Serverless::LayerVersion"
     Properties:
       ContentUri: "../../adf-build/shared/"
       CompatibleRuntimes:
-        - python3.10
+        - python3.12
       Description: "Shared Lambda Layer between master and deployment account"
       LayerName: shared_layer
 
@@ -689,8 +687,8 @@ Resources:
           phases:
             install:
               runtime-versions:
-                python: 3.11
-                nodejs: 18
+                python: 3.12
+                nodejs: 20
               commands:
                 - aws s3 cp s3://$SHARED_MODULES_BUCKET/adf-build/ ./adf-build/ --recursive --quiet
                 - pip install -r adf-build/requirements.txt -r adf-build/helpers/requirements.txt -q -t ./adf-build

src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/determine_default_branch/requirements.txt

@@ -1,2 +1,2 @@
-boto3==1.28.8
+boto3==1.34.17
 cfn-custom-resource~=1.0.1

src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/initial_commit/requirements.txt

@@ -1,4 +1,4 @@
-Jinja2==3.1.2
-boto3==1.28.8
+Jinja2==3.1.3
+boto3==1.34.17
 cfn-custom-resource~=1.0.1
 markupsafe==2.1.3

src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/pipeline_management/requirements.txt

@@ -1,4 +1,3 @@
 pyyaml~=6.0.1
 schema==0.7.5
-tenacity==8.2.2
-wrapt==1.14.1 # https://github.com/aws/aws-lambda-builders/issues/302
+tenacity==8.2.3