AWS log forwarder CloudFormation template intermittently fails to deploy because it can't get the role ARN

[JohnLudlow]

Describe what happened:
On a new environment deployment of our datadog-forwarder failed with the following error:

Unable to retrieve Arn attribute for AWS::IAM::Role, with error message The role with name datadog-forwarder-ForwarderRole-xxxxx cannot be found. (Service: Iam, Status Code: 404, Request ID: <guid>)

In the CloudFormation events we can see the ForwarderRole creation succeeded, and the Forwarder creation failed saying it couldn't find the role.

Describe what you expected:
I expected the deployment to complete successfully

Steps to reproduce the issue:

1. Prepare an empty AWS account

2. Prepare a deployment as described in the docs.

    // Pulumi C# example based on our deployment, which was based on the above docs
   
    var cloudFormationStack = new PulumiAwsCloudFormation.Stack(
        "datadog-forwarder",
        new()
        {
            Name = "datadog-forwarder",
            Capabilities = { "CAPABILITY_IAM", "CAPABILITY_NAMED_IAM", "CAPABILITY_AUTO_EXPAND" },
            TemplateUrl = "https://datadog-cloudformation-template.s3.amazonaws.com/aws/forwarder/latest.yaml",
   
            Parameters =
            {
                ["DdApiKeySecretArn"]   = dataDogApiKeySecretArn,
                ["DdSite"]              = "datadoghq.eu",
                ["DdTags"]              = $"env:{Configuration.Company}-{Configuration.Environment},envkey:{Configuration.EnvKey},version:{Configuration.LambdaVersion},release:{Configuration.Release}",
                ["DdFetchLambdaTags"]   = "true",
                ["DdFetchLogGroupTags"] = "true",
                ["FunctionName"]        = "datadog-forwarder",
            }
        }
    );
   

3. Run the deployment

4. Observe the failure

Workaround

• Destroy the stack and redeploy. The issue is intermittent so might work the second time

IMPORTANT

This is highly intermittent. You may need to try this a few times to reproduce it.

Notes from investigation

We noticed the template has this element:

We think this should depend on ForwarderRole

Resources:
  Forwarder:
    Type: AWS::Lambda::Function
    DependsOn: 
      - ForwarderZipReady
      - ForwarderRole