AIP-47 - Migrate s3 DAGs to new design #22438

[chethanuk]

No description provided.

[josh-fell]

CC @ferruzzi @vincbeck @o-nikolas I know all of you have been doing a ton of work standardizing the Amazon provider including the example DAGs.

[vincbeck]

Whao! Thanks for doing that @chethanuk-plutoflume . One general comment would be, we would like each sample dag to be self contained, meaning each sample dag creates and deletes all the resources/infrastructure it is using. One example would be in example_dynamodb_to_s3.py to create the S3 bucket and Dynamo table as setup and remove them as teardown. A good example is #24058

[o-nikolas]

Thanks for this contribution! Vincent's comment above is the most important feedback (also be sure to use the env_id in the names of the resources you create during setup so that they're unique).

Also another small nit is that we like to annotate the final task chain with which tasks are test setup, test body and test teardown. The Athena example shows this as a good example.

[chethanuk]

create the S3 bucket and Dynamo table as setup and remove them as teardown

Let's create separate PR, it might be out of scope for this PR? Since we just focusing on AIP-47?

[ferruzzi]

Let's create separate PR, it might be out of scope for this PR? Since we just focusing on AIP-47?

I'd disagree that it is out of scope. If this gets merged right now, someone is still going to have to go and make these tests actually runnable the same way as the 19+ others which have already been converted.