Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ETL-384] Add production s3 ingestion bucket #26

Merged
merged 6 commits into from
Mar 21, 2023
Merged

[ETL-384] Add production s3 ingestion bucket #26

Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
f47eb15
Add production s3 ingestion bucket
thomasyu888 Mar 21, 2023
1fd47d5
Add in s3 ingest
thomasyu888 Mar 21, 2023
64cc6d8
Remove sceptre user data
thomasyu888 Mar 21, 2023
401c54d
Remove Owner email
thomasyu888 Mar 21, 2023
57f8858
Add name of bucket
thomasyu888 Mar 21, 2023
a592f8f
Apply suggestions from code review
thomasyu888 Mar 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions config/prod/s3-ingestion.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# NOTE:
# This is the first bucket in the ingress pipeline for RECOVER project.
template:
type: "http"
url: "https://raw.githubusercontent.com/Sage-Bionetworks/aws-infra/v0.5.0/templates/S3/synapse-external-bucket.j2"
thomasyu888 marked this conversation as resolved.
Show resolved Hide resolved
stack_name: "recover-ingestion-prod"
thomasyu888 marked this conversation as resolved.
Show resolved Hide resolved
thomasyu888 marked this conversation as resolved.
Show resolved Hide resolved
stack_tags:
# This is the shared account (credentials shared on Last Pass) for RECOVER
CostCenter: "MGH RECOVER / 122500"
parameters:
# (Optional) Name of the created bucket.
BucketName: "recover-ingestion-prod"
thomasyu888 marked this conversation as resolved.
Show resolved Hide resolved
# (Optional) true for read-write bucket, false (default) for read-only bucket
AllowWriteBucket: 'true'
# (Optional) Allow accounts, groups, and users to access bucket (default is no access).
GrantAccess:
- 'arn:aws:iam::325565585839:root' # Required ARN for a synapse bucket
- 'arn:aws:sts::526515999252:assumed-role/AWSReservedSSO_S3ExternalCollab_40c062f682e7f3f5/[email protected]'
Copy link
Contributor

@rxu17 rxu17 Mar 21, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a question, does having this section in the external template just guarantee that the bucket is automatically public access? We had to do something like this for recover buckets to update to non-public access: https://github.com/Sage-Bionetworks/recover/pull/15/files

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great question @rxu17, could you help me ask IT this question?

Copy link
Contributor

@rxu17 rxu17 Mar 21, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Per this discussion here it doesn't seem like the template would block public access automatically.
We can either:

  • wait until April 2023 (not sure what date) when the bucket created initially will have the public access automatically blocked
  • Just create the bucket now, and then modify the public bucket policy on the AWS console (and this should be permanent)

Then later we could do one of the below:

  • Request an update to the external j2 template to add the configurations needed to block public access
  • Update this to use our recover s3 template yaml instead, and update our yaml to do all the things we want it to do such as grant access, etc.

Copy link
Member Author

@thomasyu888 thomasyu888 Mar 21, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rxu17 Is this true: https://sagebionetworks.slack.com/archives/CEFQD0KU1/p1679437439776059?thread_ts=1679430261.246839&cid=CEFQD0KU1. It seems we don't need to specify it.

I think the quickest path forward is to modify the bucket policy on the AWS console.

Copy link
Contributor

@rxu17 rxu17 Mar 21, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From my experience, it may only disables the ACL specific settings (I don't remember) under the public access portion. We still had to do this: https://github.com/Sage-Bionetworks/recover/pull/15/files even after our bucket had the ObjectOwnership: BucketOwnerEnforced set to block all of the public access points.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with the quickest way

- 'arn:aws:iam::621233246578:role/EES3-RK-6A32F45B-RECOVER' # QA ARN role for Care Evolution
thomasyu888 marked this conversation as resolved.
Show resolved Hide resolved