[ETL-414] Add script to generate test S3 event notification #44
Conversation
| default="main/2023-01-12T22--02--17Z_77fefff8-b0e2-4c1b-b0c5-405554c92368", | ||
| help=("Optional. Name of the Synapse dataset containing test data. " | ||
| "Default is `main/2023-01-12T22--02--17Z_77fefff8-b0e2-4c1b-b0c5-405554c92368` ")) | ||
| parser.add_argument("--input-key-prefix", |
There was a problem hiding this comment.
I just realize for recover data, if we reprocess, we can only reprocess everything at this point or if we somehow know the specific filename? Are the individual recover zipped files grouped by anything in particular other than the datetime it was created?
There was a problem hiding this comment.
Are the individual recover zipped files grouped by anything
That's a discussion we could have with digital health. They are the ones copying the data from the ingestion bucket to the input bucket, so we could come up with a partitioning structure.
In practice we would only ever submit the entire dataset. We need to remove deleted samples and those could appear anywhere.
src/lambda_function/README.md
Outdated
There was a problem hiding this comment.
Are you planning on adding a bootstrap cron trigger that does these event triggers?
There was a problem hiding this comment.
No. Are you wondering how we will submit these events as part of our tests or as part of the production pipeline?
There was a problem hiding this comment.
Thanks @philerooski. ^ Both
This script can generate a S3 event notification test event for a single S3 object or for all S3 objects which exist directly under a specific S3 key prefix. Using this test event with our lambda simulates an object create/update event in an S3 bucket.