[ApiGateway] RestApi updates account level role used for ApiGateway CloudWatch logging

[lanefelker]

This code in RestApi

aws-cdk/packages/@aws-cdk/aws-apigateway/lib/restapi.ts

Line 485 in 71aa4b6

const resource = new CfnAccount(this, 'Account', {

will update the account level role for cloudwatch logging used for all ApiGateways.

The problem we are seeing is that each new API we create will replace the role used for the account with the new role created.

If the stack that last updated the account level role gets deleted for some reason then the account level role will no longer exist and all apigateway cloudwatch logging is broken for the account 😱

Reproduction Steps

1. create a new RestApi without passing a cloudWatchRole prop
2. Deploy the new API - see the account level role change to the role associated with this new API
3. Delete the stack
4. All account level API logging no longer works because the role is deleted.

What did you expect to happen?

I would expect each apigateway logging role to be only used for a given API Gateway

or I would want the apigateway account level role to be rolled back to the previous role on deletion

What actually happened?

Described above

Environment

• CLI Version :
• Framework Version:
• Node.js Version:
• OS :
• Language (Version):

Other

Is passing a role into each API the best option to resolve this?

---

This is 🐛 Bug Report

[nija-at]

Indeed this is a broken experience.

I would expect each apigateway logging role to be only used for a given API Gateway

or I would want the apigateway account level role to be rolled back to the previous role on deletion

Unfortunately, neither of these options will work. The APIGateway service does not provide a way to specify individual roles per API endpoint. The CDK is a stateless system and does not know what the previously value here is.

I recommend setting cloudWatchRole to false in all your APIs, and explicitly using CfnAccount to set one role per AWS account.

Apologies for the poor experience. We'll need to see how to re-adjust the RestApi construct to make this experience better.

[gsdwait]

We keep running into this issue. Why should the default action be to create the role? Why not set the default for cloudWatchRole to false in RestApi and LambdaRestApi? That would mean in unusual case where you are only deploying a single rest api in an account you would simply set the cloudWatchRole to true.

[amirmohsen]

I agree with @gsdwait. The current default behaviour of having this option set to true is rather problematic as it overrides the global setting which applies to all APIs. This has caused a lot of headache for us in our company. Our team shares an AWS account with many other teams and suddenly our stack (which is the only CDK one currently) started overriding this role and having an impact on other teams. Please consider changing the default to false.

[straygar]

Any chance this could be made the default behavior for CDK v2? There have been so many times I accidentally and silently broke logs for all my API Gateways this way.

[adrian-skybaker]

Just to echo this - have we missed the window to fix this for v2?

Really hoping this isn't going to be a replay of #7140 (comment) ( "Since the API has been out there for a long time and our API Gateway module is stable, we cannot change this default").

[johnnynanjiang]

Hi @nija-at , would you reckon the simplest option for now, before a proper solution come up in an uncertain period of time, would be that the role can be retained in the function below?

  protected _configureCloudWatchRole(apiResource: CfnRestApi) {
    const role = new iam.Role(this, 'CloudWatchRole', {
      assumedBy: new iam.ServicePrincipal('apigateway.amazonaws.com'),
      managedPolicies: [iam.ManagedPolicy.fromAwsManagedPolicyName('service-role/AmazonAPIGatewayPushToCloudWatchLogs')],
    });

    this.cloudWatchAccount = new CfnAccount(this, 'Account', {
      cloudWatchRoleArn: role.roleArn,
    });

    this.cloudWatchAccount.node.addDependency(apiResource);
  }

[sekrupke]

This behaviour is indeed very annoying and may cause that devs search for the API logs and wondering why them are nowhere to be found. Would be great if that could be fixed.

[andrei-cdl]

Same here, we has this happen in our environment and silently changed the behaviour. Thankfully it was a side effect and this issue is open. There's probably a few teams out there not even realizing this behaviour is happening in their infrastructure.

[jpSimkins]

I just noticed this issue today as well... We've had this issue for over a year without even knowing it... yup, one of those teams that wasn't aware of this bug feature.

This affects PROD systems, I really hope this get addressed. I am testing if cloudWatchRole to false will solve our issue. Either way, defaulting to an account wide CloudWatch role seems like a very bad decision. I really hope this gets fixed but given how old this ticket is... I am doubtful.

Also, setting a single CloudWatch Role for the entire account is not ideal and raises many other concerns. I'm surprised this is recommended.

An example of this breaking a PROD system:

• Deploy an API Gateway using LambdaRestApi from the api-gateway module.
• Assign a role to the lambda function
• Deploy

Now all API Gateways are using the role for the lambda function! This is dangerous for the default behavior.

Update:

Yes, using cloudWatchRole: false, stops this issue. Why is this is not the default?

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.