aws-cdk/hotswap: The bootstrap roles don't contain necessary permissions for hotswapping

[ajhool]

Describe the bug

When we tried to add the --hotswap flag to our deployment command in CICD development environments, the deployment failed:

❌ Deployment failed: AccessDenied: User: arn:aws:sts::***:assumed-role/github-actions-oidc-deployment-role/cicd is not authorized to perform: cloudformation:DescribeStacks on resource: arn:aws:cloudformation:us-east-1:***:stack/***/2765aac0-eb87-11ed-8a91-0ad92641443b because no identity-based policy allows the cloudformation:DescribeStacks action

The hotswap developer comments that the standard CDK bootstrap roles don't contain permissions to update Lambda functions directly (and presumably all of the other necessary hotswappable resource types), so they use some other credentials for the SDK:

aws-cdk/packages/aws-cdk/lib/api/hotswap-deployments.ts

Line 54 in a65cddf

// create a new SDK using the CLI credentials, because the default one will not work for new-style synthesis -

Expected Behavior

We should be able to add the hotswap flag while using the CDK deployed bootstrap roles:

cdk bootstrap
cdk deploy
# No action needed related to IAM, the following should just work
cdk deploy --hotswap

Current Behavior

cdk bootstrap

Using an IAM user with these permssions that just allow the IAM user to assume the CDK bootstrap roles:

    /**
     * DeploymentRole that is assumed by GithubAction using OIDC. This role enables
     * GitHub action to interact with AWS resources.
     */
    this.deploymentRole = new Role(this, `deployment-role`, {
      roleName: `github-actions-oidc-deployment-role`,
      assumedBy: new FederatedPrincipal(
        this.oidcProvider.openIdConnectProviderArn,
        oidcConditions,
        "sts:AssumeRoleWithWebIdentity",
      ),
    });

    /**
     * This policy is able to assume cross-account roles using the
     * roles created by the CDK Bootstrap command. The `cdk` commands will
     * choose the appropriate role for the `cdk` subcommand.
     */
    this.assumeCdkPolicy = new Policy(this, "policy", {
      statements: [
        new PolicyStatement({
          effect: Effect.ALLOW,
          actions: ["sts:AssumeRole"],
          resources: ["arn:aws:iam::*:role/cdk-*"],
        }),
      ],
      roles: [this.deploymentRole],
    });

cdk deploy // This works
cdk deploy --hotswap // This fails due to the stack error from before

Reproduction Steps

cdk bootsrap

// Initially any CDK app
cdk init app --language typescript

// Add a NodejsFunction to the CDK sample stack because that is hotswappable

// Create IAM user with the "AssumeRole" "arn:aws:iam::*:role/cdk-*" policy above
// Using the IAM user created above...

cdk deploy; // No problems

// Make a change to the lambda to trigger hotswapping

cdk deploy --hotswap

// Expect error related to DescribeStacks

Possible Solution

The bootstrap command should create a more permissive hotswap deployment role. An "enable-hotswap-role" or "disable-hotswap-role" flag could be used to toggle the creation of that role -- I'm not sure which is a more sensible default.

As a short-term alternative, documentation of the required permissions on the CLI role would be helpful.

Additional Information/Context

No response

CDK CLI Version

2.86.0

Framework Version

No response

Node.js Version

20.1.0

OS

Linux / Github Actions

Language

Typescript

Language Version

No response

Other information

No response

[khushail]

Hi @ajhool , thanks for reaching out. It looks like your issue is bit similar to this previous raised issue and has a submitted PR as well. Feel free to reach out if it looks different

[ajhool]

Hi @khushail I don't believe that issue (#26785) addresses or is directly related to this issue. I don't see any permissions related details in that issue or PR. That issue appears to be about adding support for hotswap to a different CDK interface (the library, as opposed to the cli).

I'm not sure how the cdk-lib manages roles, but if it uses the same role management approach as CDK CLI I would expect that library to also be affected by this issue.

I will make a comment on that pointing this out and maybe they'll be able to include hotswap Role support in that PR.

[khushail]

@ajhool , My bad, I did not get enought time to look into detail of that issue and linked it.
Thanks for taking that note and taking further step to mention it. For now, I would be marking this as P2.

[ajhool]

Thanks @khushail . For anybody who might come across this issue the (hopefully temporary) workaround was to add AdministratorAccess to the IAM user that the github action is assuming. I'm only using hotswap in sandbox accounts, so I don't mind enabling AdministratorAccess access to those accounts. When support is added to the bootstrap roles, I'll remove the AdministratorAccess policy from that role.

[dsolowitz]

When trying to manually assume a role using cdk --role-arn deploy --hotswap, this doesn't actually assume the role. It still defaults to the user associated with the configuration.
Also, CDK watch, "appears" to be working (no errors in console) however, when saving changes, those changes aren't applied to the lambda... I assume this is related.