codepipeline: x-account AND x-region deployments are a mess

[rix0rrr]

Describe the bug

A cross-region AND cross-account deployment where CodePipeline generates the support stacks for replication buckets and the support stacks for cross-account roles (which is the ideal situation), fails to deploy out of the box.

The cross-account and cross-region resources are created in separate stacks that don't have a dependency between them, but the resources on those stacks have bidirectional dependencies on each other:

• The Bucket and Key policies reference the Action Role
• The Role policy references the Bucket and Key (and deployment resources like CodeDeploy Deployment Groups)

Normally this wouldn't deploy, but we generate hard-coded resource names for the resources involved so that we can formulate policies anyway without having to have bidirectional cross-stack references.

---

Everything we're doing in this area right now is a patchwork of ad-hoc solutions, that needs to be rethought.

The bidirectional policy references between the support stacks should be replaced with some form of tag-based access control and clean dependencies between the stacks (and not the "we put some policies in both and hope it works" style we do today).

We should have a good think on what resources go into what stacks in what cases, keeping in mind:

• The deployment resources can be owned or referenced (what stack are they int?)
• The action role can be owned or referenced (what stack are they in, what if the user created them?)
  • If this goes into a support stack, what region does that support stack deploy into? us-east-1, $pipeline-region or $resources-region? Why is that the right choice?
• The replication resources can be owned or referenced (what stack are they in, what if the user created them?)

Someone needs to go look at the current situation, come up with a good design, and cover the delta.

Expected Behavior

Cross-environment deployments to work with a minimum of fuss.

Current Behavior

Hard-to-debug failures or mysterious cyclic dependencies between stacks.

[ldecaro]

If there's a pipeline with a cross-region action and if this action needs to transfer files from one region to the other, the cross-region-stack will be created. In my use case, this is a CodeDeploy action. It requires an action role, that CDKPipelines may generate or you can use your own role. No matter what you choose, cdk will throw an error like the one below:

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:3.1.0:java (default-cli) on project service: An exception occurred while executing the Java class. Error: 'DemoToolchain' depends on 'cross-region-stack-111111111111:us-east-2' ({DemoToolchain}.addDependency({cross-region-stack-111111111111:us-east-2})). Adding this dependency (replication bucket {cross-region-stack-111111111111:us-east-2/Default/CrossRegionCodePipelineReplicationBucket} to action role {DemoToolchain/AWSCodeDeployRoleForBlueGreen-UAT}) would create a cyclic reference.

[ERROR] @jsii/kernel.RuntimeError: Error: 'DemoToolchain' depends on 'cross-region-stack-111111111111:us-east-2' ({DemoToolchain}.addDependency({cross-region-stack-111111111111:us-east-2})). Adding this dependency (replication bucket {cross-region-stack-111111111111:us-east-2/Default/CrossRegionCodePipelineReplicationBucket} to action role {DemoToolchain/AWSCodeDeployRoleForBlueGreen-UAT}) would create a cyclic reference.

This problem will not happen if the deployment is cross-account (single or cross-region). If I have a pipeline that deploys a Service using CodeDeployAction in any region other than the region where the pipeline is deployed, the error above shows.