[ETL-621] Add SNS topic to broadcast new exports #106
Conversation
|
There was a problem hiding this comment.
LGTM! Were some of the main reasons for us moving from S3 -> SQS to S3 -> SNS -> SQS due to being able to decouple our S3 bucket from SQS (given we have multiple now) and make it easier to manage (we don't have to modify S3 bucket configuration directly as much if anything changes on the SQS queue side).
No, it's not possible to have S3 send event notifications to more than one destination for the same S3 prefix. I'm not sure why this is, but maybe they don't want to have to support SNS-like capabilities for every service they implement event hooks for. |
I thought we could configure multiple notification configs to the S3 bucket with the same s3 prefix. Guessing that doesn't execute. Bummer. |
| return sqs_event_record | ||
|
|
||
| def create_sns_notification(s3_event_record): |
Lot's of files touched here, but essentially I've inserted an SNS topic between our S3 input bucket and our SQS queues (now there's two queues!) in anticipation of triggering both input -> intermediate and input -> raw processes in reaction to new export data. That looks sort of like this:
templates/sqs-queue.yamlwas modified to subscribe to the SNS topicsrc/lambda_function/s3_to_glue/app.pywas modified to work with events formatted asSQS(SNS(S3))config/prod/namespaced/s3-event-config-lambda.yamlwas modified to add the S3 event notification for the SNS topic, rather than the SQS queue.Renamed files:
config/develop/namespaced/s3-event-config-lambda.yamlrenamed toconfig/develop/namespaced/lambda-s3-event-config.yamlto match the pattern{service_name}-{descriptor}config/develop/namespaced/s3-event-config-lambda-role.yamlrenamed toconfig/develop/namespaced/lambda-s3-event-config-role.yamlprodNew files:
config/develop/namespaced/sns-topic.yamlconfig/prod/namespaced/sns-topic.yamltemplates/sns-topic.yaml