aws-certificatemanager: invalid cloudformation generated when using wildcard domain to create public certificate

[alexbaileyuk]

Describe the bug

When creating a public certificate with a wildcard subdomain, invalid Cfn template is generated and stack creation fails. Cfn includes the same domain validation records twice with exactly the same values which fails when sending the DNS change batch.

Edit: after further investigation, this happens when importing SSM parameters from cross region using the suggested workaround here: https://github.com/henrist/cdk-cross-region-params/blob/master/src/parameter-reader.ts or https://stackoverflow.com/questions/71246435/how-to-read-parameter-store-from-a-different-region-in-cdk. This is a fairly well used way of getting around restrictions with cross-region exports etc.

Expected Behavior

When creating a certificate with wildcard DNS, the correct Cfn is generated and can be deployed.

Expected Cfn output (slightly edited to avoid private information):

  "Certificate4E7ABB08": {
   "Type": "AWS::CertificateManager::Certificate",
   "Properties": {
    "DomainName": {
     "Fn::GetAtt": [
      "ZoneNameReaderFunctionCBCCE744",
      "Parameter.Value"
     ]
    },
    "DomainValidationOptions": [
     {
      "DomainName": {
       "Fn::GetAtt": [
        "ZoneNameReaderFunctionCBCCE744",
        "Parameter.Value"
       ]
      },
      "HostedZoneId": {
       "Fn::GetAtt": [
        "ZoneIdReaderFunction84D6D92A",
        "Parameter.Value"
       ]
      }
     }
    ],
    "SubjectAlternativeNames": [
     {
      "Fn::Join": [
       "",
       [
        "*.",
        {
         "Fn::GetAtt": [
          "ZoneNameReaderFunctionCBCCE744",
          "Parameter.Value"
         ]
        }
       ]
      ]
     }
    ],
    "Tags": [
     {
      "Key": "Name",
      "Value": "PipelineStack/TestEnvironmentStacksDeployment/CertificateStack-eu-west-1/Certificate"
     }
    ],
    "ValidationMethod": "DNS"
   },
   "Metadata": {
    "aws:cdk:path": "PipelineStack/TestEnvironmentStacksDeployment/CertificateStack-eu-west-1/Certificate/Resource"
   }
  }

Current Behavior

When creating a certificate with wildcard DNS, the incorrect Cfn is generated and can be deployed.

Example Cfn output (slightly edited to avoid private information):

  "Certificate4E7ABB08": {
   "Type": "AWS::CertificateManager::Certificate",
   "Properties": {
    "DomainName": {
     "Fn::GetAtt": [
      "ZoneNameReaderFunctionCBCCE744",
      "Parameter.Value"
     ]
    },
    "DomainValidationOptions": [
     {
      "DomainName": {
       "Fn::GetAtt": [
        "ZoneNameReaderFunctionCBCCE744",
        "Parameter.Value"
       ]
      },
      "HostedZoneId": {
       "Fn::GetAtt": [
        "ZoneIdReaderFunction84D6D92A",
        "Parameter.Value"
       ]
      }
     },
     {
      "DomainName": {
       "Fn::Join": [
        "",
        [
         "*.",
         {
          "Fn::GetAtt": [
           "ZoneNameReaderFunctionCBCCE744",
           "Parameter.Value"
          ]
         }
        ]
       ]
      },
      "HostedZoneId": {
       "Fn::GetAtt": [
        "ZoneIdReaderFunction84D6D92A",
        "Parameter.Value"
       ]
      }
     }
    ],
    "SubjectAlternativeNames": [
     {
      "Fn::Join": [
       "",
       [
        "*.",
        {
         "Fn::GetAtt": [
          "ZoneNameReaderFunctionCBCCE744",
          "Parameter.Value"
         ]
        }
       ]
      ]
     }
    ],
    "Tags": [
     {
      "Key": "Name",
      "Value": "PipelineStack/TestEnvironmentStacksDeployment/CertificateStack-eu-west-1/Certificate"
     }
    ],
    "ValidationMethod": "DNS"
   },
   "Metadata": {
    "aws:cdk:path": "PipelineStack/TestEnvironmentStacksDeployment/CertificateStack-eu-west-1/Certificate/Resource"
   }
  }

Both of the DomainValidationOptions generate the same validation CNAME record. When the call to put records is done, it includes a duplicate record. See the below CloudTrail for example:

"errorCode": "InvalidChangeBatch",
    "errorMessage": "[The request contains an invalid set of changes for a resource record set 'CNAME _00e851c0ad3747082dea8abddd0f1515.test.my-domain.com.']",
    "requestParameters": {
        "hostedZoneId": "MY_ZONE_ID",
        "changeBatch": {
            "changes": [
                {
                    "action": "UPSERT",
                    "resourceRecordSet": {
                        "name": "_thecnamegenerated.test.my-domain.com.",
                        "type": "CNAME",
                        "tTL": 300,
                        "resourceRecords": [
                            {
                                "value": "_thecnamegenerated.mrhcxwpsky.acm-validations.aws."
                            }
                        ]
                    }
                },
                {
                    "action": "UPSERT",
                    "resourceRecordSet": {
                        "name": "_thecnamegenerated.test.my-domain.com.",
                        "type": "CNAME",
                        "tTL": 300,
                        "resourceRecords": [
                            {
                                "value": "_thecnamegenerated.mrhcxwpsky.acm-validations.aws."
                            }
                        ]
                    }
                }
            ]
        }
    },

Reproduction Steps

import { Stack, StackProps } from 'aws-cdk-lib';
import { Construct } from 'constructs';
import { HostedZone } from 'aws-cdk-lib/aws-route53';
import { Certificate, CertificateValidation } from 'aws-cdk-lib/aws-certificatemanager';

export class CertificateStack extends Stack {
  constructor(scope: Construct, id: string, props: StackProps) {
    super(scope, id, props);

    const zoneId = new SSMParameterReader(this, 'ZoneId', {
      parameterName: DomainStack.HOSTED_ZONE_ID_PARAMETER,
      region: 'us-east-1'
    }).getParameterValue();

    const zoneName = new SSMParameterReader(this, 'ZoneName', {
      parameterName: DomainStack.HOSTED_ZONE_NAME_PARAMETER,
      region: 'us-east-1'
    }).getParameterValue();

    const zone = HostedZone.fromHostedZoneAttributes(this, 'Zone', {
      hostedZoneId: zoneId,
      zoneName: zoneName // eg. my-domain.com
    });

    const sharedCertificate = new Certificate(this, 'Certificate', {
      domainName: zoneName, // eg. my-domain.com
      subjectAlternativeNames: [`*.${zoneName}`], // eg. *.my-domain.com
      validation: CertificateValidation.fromDns(zone)
    });
  }
}

Possible Solution

Unsure the best way to deal with this to be honest. Potentially need to understand more about the scenarios that domain validation cname records can clash. When they clash, some logic may need to be executed.

The current workaround I've put in place is:

  private patchCertificateValidation(certificate: Certificate) {
    const cfnCertificate = certificate.node.defaultChild as CfnCertificate;

    if (Array.isArray(cfnCertificate.domainValidationOptions) && cfnCertificate.domainValidationOptions.length === 2) {
      delete cfnCertificate.domainValidationOptions[1];
    }
  }

Of course this works for my scenario but may need to be updated for other peoples requirements depending on what domains and alternative domains are requested. It could be improved to read all domains in the list and look specifically for wildcards that need to be removed.

Additional Information/Context

Unsure if it would help since most people won't have access to the case details but this fix was suggested by AWS Support in case 13934396421. This was roughly the response:

Upon dive deep, I found that cloudtrail request which got failed as InvalidChangeBatch - 834fb28e-8ec1-47a3-ab8c-3c83cfd8b410 - has two UPSERT actions of same record and value. Upon checking the CloudFormation template and checking with CloudFormation engineer, they have suggested to remove the wildcard value under the DomainValidationOptions.

Please keep the SubjectAlternativeNames with both Apex domain (test.my-domain.com.) and Wildcard domain (*.test.my-domain.com.). This will be able to create the ACM certificate for both the domains and certificate will also be able to get validated by DNS.

CDK CLI Version

2.94.0

Framework Version

No response

Node.js Version

v18.12.1

OS

Ubuntu 22.04.2 LTS

Language

Typescript

Language Version

4.9.5

Other information

Also tested on the latest version of the CDK (2.99.1) with the same outcome.

Probably related to #15574.

Only seems to be an issue using HostedZone.fromHostedZoneAttributes. Other code paths appear to work. Also noted in #9248.

This was supposedly fixed in https://github.com/aws/aws-cdk/pull/9291/files but that doesn't appear to be the case.

[alexbaileyuk]

Upon closer review of the CDK code and test files it looks like this is because we're using the SSM reader and the output Cfn is using the GetAttr. Not sure if this will turn into a "won't fix" or not but lets see.

[peterwoodworth]

Ideally, these do get filtered out. However, since the values are not known at deploy time, we cannot run the necessary logic we have in place - and have to assume the deploy time value will be a viable use case.

aws-cdk/packages/aws-cdk-lib/aws-certificatemanager/lib/certificate.ts

Lines 318 to 327 in b2a895e

	/**
	* Removes wildcard domains (*.example.com) where the base domain (example.com) is present.
	* This is because the DNS validation treats them as the same thing, and the automated CloudFormation
	* DNS validation errors out with the duplicate records.
	*/
	function getUniqueDnsDomainNames(domainNames: string[]) {
	return domainNames.filter(domain => {
	return Token.isUnresolved(domain) || !domain.startsWith('*.') || !domainNames.includes(domain.replace('*.', ''));
	});
	}

I don't see a great way to implement an official way to work around this with our current API either, unless we add an entirely new prop specifically to signal that this use case is being implemented. @indrora what do you think?

Thanks for posting a workaround that works for you 🙂, that workaround should work as long as you are deleting the correct one, just be sure to check the template to see which index in the array to delete.

[vsnig]

Another workaround, If you're deploying with CICD Pipeline you actually can resolve SSM param in pipeline before deploying CDK

Github Actions pipeline:

  - name: ⚙️ Fetch SSM Parameters and set Env variables
        uses: dkershner6/aws-ssm-getparameters-action@v2
        with:
          parameterPairs: '/dns/myDomain = MY_DOMAIN'

then, in CDK:

const myDomain = process.env.MY_DOMAIN!

const cert = new acm.Certificate(this, 'cert', {
      domainName: myDomain,
      validation: acm.CertificateValidation.fromDns(zone),
      subjectAlternativeNames: [`*.${myDomain}`],
    })