(codepipeline): Cross-region support stacks break when their template exceeds 50kb

[gshpychka]

Describe the bug

Currently, cross-region support stacks use the BootstraplessSynthesizer by default, reusing the pipeline environment's default execution and deploy roles and thus avoiding the requirement to bootstrap all regions you want your actions to use.

This works fine most of the time, until their template reaches the CloudFormation size limit for including the template in a request, which is 51,200 bytes. When this happens, the template needs to be uploaded to S3 as an asset instead - and this fails.

We have seen this with CDK Pipelines, and what happens in practice is that the SelfMutation Action tries to upload the template to the default asset bucket in the target region, but fails to do so (as it should, because it lacks the access to do that).

Expected Behavior

Pipeline works fine.

Current Behavior

Pipeline tries and fails to upload the support stack template to the asset bucket in the target region.

Reproduction Steps

Deploy a CDK Pipeline that deploys to a lot of accounts in another region such that the support stack reaches 50kb.

Possible Solution

Generate a descriptive error when a cross-region support stack reaches 50Kb. Add a flag that would allow us to control the synthesizer to be used in support stacks (or just an option to use the parent one).

I would be glad to open a PR, but would need some guidance on how to detect the case.

Additional Information/Context

A very hacky workaround is to add explicit statements allowing S3 access to the asset bucket in the target region to the SelfMutation Role, as well as the CDKToolkit's deploy role in the local region. The target region must be bootstrapped.

CDK CLI Version

2.94.0

Framework Version

2.94.0

Node.js Version

18.4.1

OS

MacOS

Language

Typescript, Python, .NET, Java, Go

Language Version

No response

Other information

No response

[peterwoodworth]

Thanks for the report @gshpychka