aws-opensearchservice: Domain creates new ResourcePolicy if logging enabled

[andresblancomorales]

Describe the bug

Creating a new OpenSearch domain creates a new ResourcePolicy every time if any of: slowSearchLogEnabled, slowIndexLogEnabled, appLogEnabled, auditLogEnabled is true.

According to the CloudWatch limits:

Up to 10 CloudWatch Logs resource policies per Region per account. This quota can't be changed.

We're hitting this quota easily since every domain we create creates a new ResourcePolicy

Expected Behavior

Not sure what what would be a good expected behavior.

May be: Reuse a previously created ResourcePolicy and add the new log policy statement.

Current Behavior

A new resource policy is created with every new domain that is created with logging enabled.

Reproduction Steps

Instantiate a new Domain and set any of the log types as true:

                 const domain = new Domain(stack, 'TheDomain', {
                      version: EngineVersion.OPENSEARCH_1_2,
                      logging: {
                          appLogEnabled: true
                      }
                  })

Possible Solution

Reuse a previously created ResourcePolicy and add the new log policy statement.

Additional Information/Context

No response

CDK CLI Version

1.187.0

Framework Version

1.187.0

Node.js Version

16.19.0

OS

Linux

Language

Typescript

Language Version

4.4.3

Other information

No response

[pahud]

related to #22307

Yes we are aware of the resource policy number limit and we definitely should have a workaround for that.

[keenangraham]

This is an issue for us too.

We run into a hard AWS limit of ten Opensearch domains per region when logging is enabled, preventing us from spinning up as many demos as we need for development.

https://docs.aws.amazon.com/opensearch-service/latest/developerguide/createdomain-configure-slow-logs.html#createdomain-configure-slow-logs-cli says:

Important
CloudWatch Logs supports 10 resource policies per Region. If you plan to enable slow logs for several OpenSearch Service domains, you should create and reuse a broader policy that includes multiple log groups to avoid reaching this limit.

Looking at the CDK code a new CustomResource LogGroupResourcePolicy is created automatically for every domain (when any logging is enabled) without the ability to prevent its creation or use a broader policy:

aws-cdk/packages/aws-cdk-lib/aws-opensearchservice/lib/domain.ts

Lines 1492 to 1507 in c224fc3

	if (logGroups.length > 0) {
	const logPolicyStatement = new iam.PolicyStatement({
	effect: iam.Effect.ALLOW,
	actions: ['logs:PutLogEvents', 'logs:CreateLogStream'],
	resources: logGroups.map((lg) => lg.logGroupArn),
	principals: [new iam.ServicePrincipal('es.amazonaws.com')],
	});
	// Use a custom resource to set the log group resource policy since it is not supported by CDK and cfn.
	// https://github.com/aws/aws-cdk/issues/5343
	logGroupResourcePolicy = new LogGroupResourcePolicy(this, `ESLogGroupPolicy${this.node.addr}`, {
	// create a cloudwatch logs resource policy name that is unique to this domain instance
	policyName: `ESLogPolicy${this.node.addr}`,
	policyStatements: [logPolicyStatement],
	});
	}

Would it be possible to use a previously created broader resource policy per the suggestion in the docs, or somehow stop the CDK from trying to create a new one if logs are enabled?

[XiaowenMaoA]

what's the workaround here?

[sakurai-ryo]

@XiaowenMaoA
Could you try this code as a workaround until the PR is merged?

    const domain = new opensearch.Domain(this, 'Domain', domainProps);
    const domainResource = domain.node.defaultChild as opensearch.CfnDomain;
    domainResource.addOverride('DependsOn', undefined);

    domain.node.children
      .filter(child => child instanceof AwsCustomResource)
      .forEach(value => domain.node.tryRemoveChild(value.node.id));

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.