aws-s3: Bucket with notifications has a delete stack race condition

[adrianmace]

Describe the bug

When you define a Bucket resource and attach notifications to it (using the Custom::S3BucketNotifications resource under the hood) then try to delete the whole stack, it can sometimes (often) get into a race condition where the bucket delete and final put_bucket_notification_configuration call conflict.

A picture is worth 1000 words.

Expected Behavior

The stack can be deleted successfully.

Current Behavior

The delete fails. See image above.

If you click delete again, it succeeds the second time.

Reproduction Steps

The code above is programmatically generated and cannot be shared. The crux of the issue is however to:

1. Create a Lambda function
2. Create a bucket bucket
3. Attach a notification on the bucket for ObjectCreated Put
4. Attach a notification on the bucket for ObjectCreated Post (important: two notifications configured)
5. Deploy the stack
6. Attempt to delete the stack

Possible Solution

Make the function dependant on the bucket, so that Cloudformation will delete the function before the bucket in every situation.

Additional Information/Context

No response

CDK CLI Version

2.59.0

Framework Version

No response

Node.js Version

N/A

OS

N/A

Language

Python

Language Version

Python 3.10.9

Other information

No response

[pahud]

Thank you for the report. Are you able to share a repo with minimal required cdk code so we can deploy in our account and reproduce that?

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[adrianmace]

Hi @pahud,

Sorry that I didn't get back to you on this.

It looks like #24762 is possibly related to this / a duplicate. That was closed and merged into #16811 as being a duplicate, but overall this 'race condition' still exists.

I don't have a minimal example handy since this is built into a pretty complicated CDK app, but the reproduction steps are complete.

I can confirm this is still an issue which is causing us trouble at least weekly.