aws-s3-deployment: Source.jsonData fails for certain types of deploy-time values

[ktmayes-amzn]

Describe the bug

Some types of deploy-time values cause Source.jsonData (and presumably the other deploy-time static methods for Source) to fail since these deploy-time values contain CloudFormation functions other than Ref or Fn::Get.

For example, the secretName field on a Secret becomes an Fn::Join function call which gets rejected by Source.jsonData (see Reproduction Steps for example code).

Expected Behavior

For the rendered JSON file containing the secret ID to be deployed to the S3 bucket.

Current Behavior

For the code for reproducing the issue, I get this error:

Error: Invalid CloudFormation reference. "Ref" or "Fn::GetAtt". Got {"Fn::Join":["-",[{"Fn::Select":[0,{"Fn::Split":["-",{"Fn::Select":[6,{"Fn::Split":[":",{"Ref":"testsecretF8BBC644"}]}]}]}]},{"Fn::Select":[1,{"Fn::Split":["-",{"Fn::Select":[6,{"Fn::Split":[":",{"Ref":"testsecretF8BBC644"}]}]}]}]}]]}

Reproduction Steps

const bucket = new Bucket(this, 'test-bucket');
const secret = new Secret(this, 'test-secret');
new BucketDeployment(this, 'test-bucket-deployment', {
  destinationBucket: bucket,
  sources: [Source.jsonData('key.json', { secretId: secret.secretName })],
});

Possible Solution

No response

Additional Information/Context

Seems to be rejected by this code:

aws-cdk/packages/aws-cdk-lib/aws-s3-deployment/lib/render-data.ts

Lines 61 to 73 in dde8a4a

	function addMarker(part: Ref | GetAtt) {
	const keys = Object.keys(part);
	if (keys.length !== 1 || (keys[0] != 'Ref' && keys[0] != 'Fn::GetAtt')) {
	throw new Error(`Invalid CloudFormation reference. "Ref" or "Fn::GetAtt". Got ${JSON.stringify(part)}`);
	}
	const marker = `<<marker:0xbaba:${markerIndex++}>>`;
	result.push(marker);
	markers[marker] = part;
	}
	return { text: result.join(''), markers };
	}

CDK CLI Version

2.75.0 (build 37c53d6)

Framework Version

No response

Node.js Version

18

OS

macOS

Language

Typescript

Language Version

TypeScript (4.9.5)

Other information

No response

[pahud]

Thanks for your report. Can you share your content of key.json so I can rapidly reproduce it in my account?

[pahud]

And you are right. The addMarker() seems only support Fn::GetAtt and Fn::GetAtt while you were passing Fn::Join and Fn::Select which are not supported yet.

[ktmayes-amzn]

Thank you for the quick response!

I'm not sure what you mean about the contents of key.json, I think my code sample should be self-contained in that it triggers the bug when synthesizing a template for a stack containing that code.

If you want what { secretId: secret.secretName } from the code sample becomes in CloudFormation-speak, it is this:

Example CloudFormation JSON rejected by `Source.jsonData`

{
  "Fn::Join": [
    "",
    [
      "{\"secretId\":\"",
      {
        "Fn::Join": [
          "-",
          [
            {
              "Fn::Select": [
                0,
                {
                  "Fn::Split": [
                    "-",
                    {
                      "Fn::Select": [
                        6,
                        {
                          "Fn::Split": [
                            ":",
                            {
                              "Ref": "testsecretF8BBC644"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ]
            },
            {
              "Fn::Select": [
                1,
                {
                  "Fn::Split": [
                    "-",
                    {
                      "Fn::Select": [
                        6,
                        {
                          "Fn::Split": [
                            ":",
                            {
                              "Ref": "testsecretF8BBC644"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        ]
      },
      "\"}"
    ]
  ]
}

I added console.log(JSON.stringify(obj, null, " ")); to the renderData function in the aws-cdk-lib/aws-s3-deployment/lib/render-data.js file in my node_modules.

[Justinon]

Also seeing this issue:

Error: Invalid CloudFormation reference. "Ref" or "Fn::GetAtt". Got {"Fn::Select":[1,{"Fn::Split":["/",{"Ref":"SsmParameterValue<REDACTED>Parameter"}]}]}

[Justinon]

Also seeing this issue:

Error: Invalid CloudFormation reference. "Ref" or "Fn::GetAtt". Got {"Fn::Select":[1,{"Fn::Split":["/",{"Ref":"SsmParameterValue<REDACTED>Parameter"}]}]}

This comes from the following:

let tg: ApplicationTargetGroup // Assume this is defined
const str = tg.firstLoadBalancerFullName + '/' + tg.targetGroupFullName

Followed by then utilizing that str as a value in a JSON blob being passed as obj for Source.jsonData.

[Justinon]

Played around w/ my local copy of node_modules/aws-cdk-lib/aws-s3-deployment/lib/render-data.js and removed the conditional in addMarker from this:

if (keys.length !== 1 || (keys[0] != 'Ref' && keys[0] != 'Fn::GetAtt')) {

to this:

if (keys.length !== 1) {

and...it worked beautifully. The rendered value in the S3 object itself is 100% correct. Why have this condition at all? Perhaps either removing it, or adding && keys[0] != 'Fn::Select', && keys[0] != 'Fn::Join', etc. will solve this.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.