v0.2.3

What's Changed

• Add --input-key-file parameter to test event generation script by @philerooski in #68
• [ETL-517] Write JSON records as NDJSON in blocks by @philerooski in #69

Full Changelog: v0.2.2...v0.2.3

v0.2.2

What's Changed

• [ETL-495] Remove invalid property from github workflow by @philerooski in #64
• [ETL-505] Add Garmin transforms to S3 to JSON job by @philerooski in #65
• [ETL-499] JSON to Parquet support for Garmin data types by @philerooski in #66
• Add an owner.txt to the ingestion bucket by @philerooski in #67

Full Changelog: v0.2.1...v0.2.2

v0.2.1

What's Changed

• [ETL-446] Add upload-and-deploy-to-prod-main workflow by @philerooski in #55
• [ETL-464] Grant Care Evolution Ped/Adult roles access to ingestion bucket by @philerooski in #56
• [ETL-466] Archive existing parquet datasets before writing parquet by @philerooski in #57
• [ETL-333] Add tests for JSON to Parquet Glue job by @philerooski in #58
• [ETL-483] Fix testing of bucket access via STS token credentials by @rxu17 in #60
• [ETL-327] Integrate RECOVER tests into CI/CD by @rxu17 in #59
• [ETL-407] Add integration tests for prod/staging by @philerooski in #61
• [ETL-490] Delete local copy of JSON after uploading to S3 by @philerooski in #62
• [ETL-492] Update Glue table schemas to match crawler findings by @philerooski in #63

Full Changelog: 0.2.0...v0.2.1

0.2.0

What's Changed

• [ETL-435] Deny Synapse-connected bucket access outside us-east-1 by @philerooski in #49
• [ETL-447] Provide S3 access to pranav for ingestion bucket by @thomasyu888 in #52
• [ETL-397] Create s3-event-config lambda and dependencies to add s3 notification configuration by @rxu17 in #50
• [ETL-457] Include digital health service account in owner.txt by @philerooski in #53
• [ETL-457] Add digital health service account to processed bucket owner.txt by @philerooski in #54

Full Changelog: 0.1.0...0.2.0

0.1.0

What's Changed

• [ETL-280] add pipfile, .yamllint, and pre commit config files by @rxu17 in #1
• [ETL-281] add just pre-commit actions in upload-and-deploy + codeql by @rxu17 in #2
• [ETL-281] update codeql action to v2, actions to v3 by @rxu17 in #3
• [ETL-284] s3 bucket and stack creation by @rxu17 in #4
• Add cloudformation bucket stacks by @philerooski in #5
• [ETL-283] S3 to JSON job by @philerooski in #6
• [ETL-290] S3 to json lambda by @rxu17 in #7
• Stuff I forgot in ETL-283 by @philerooski in #8
• [ETL-328] Load JSON string as object for SymptomLog data type by @philerooski in #9
• [ETL-294] create glue table stack by @rxu17 in #10
• [ETL-329] delete check by @rxu17 in #12
• [ETL-332] create lambda iam role by @rxu17 in #13
• [ETL-287] JSON to Parquet job by @philerooski in #11
• [ETL-340] Set up sts access by @rxu17 in #14
• [ETL-326] block public access in bucket by @rxu17 in #15
• [ETL-341] s3 to json tests by @rxu17 in #16
• [ETL-328] Encode EnrolledParticipants JSON string field as object by @philerooski in #17
• Write type of HealthKitV2Statistics to JSON by @philerooski in #18
• Update S3 to JSON tests by @philerooski in #19
• [ETL-361] Fix lambda function to use env var due to failing test by @rxu17 in #20
• [ETL-336] Upload artifacts by @rxu17 in #21
• [ETL-336] Separate out upload artifacts actions for develop vs prod env by @rxu17 in #22
• Refactor artifacts github action (part of ETL-365) by @rxu17 in #23
• [ETL-365] Launch stacks during push and PR to dev AWS account by @rxu17 in #24
• [ETL-384] Add production s3 ingestion bucket by @thomasyu888 in #26
• [IBCDPE-534] Add develop s3 Ingestion bucket by @thomasyu888 in #25
• [ETL-385] Add in s3 production data pre-etl bucket by @thomasyu888 in #27
• [ETL-398] Add glue, python version params to relevant prod glue configs by @rxu17 in #29
• [ETL-380] Add cleanup workflow by @philerooski in #28
• Add permissions for OIDC to cleanup action by @philerooski in #30
• [ETL-395] Modify prod stack configs to use new pre-ETL bucket name by @rxu17 in #31
• [ETL-386] Rename pilot data bucket and link to Synapse by @philerooski in #32
• [ETL-394] Update prod and dev account resources to use NAMESPACE by @rxu17 in #33
• [ETL-382] Update folder structure of dev resources by @rxu17 in #34
• [ETL-401] Update dependencies for stack configs in prod and dev accounts by @rxu17 in #35
• [ETL-386] Grant Synapse access to dev pre-ETL bucket by @philerooski in #36
• [ETL-381] Grant Synapse access to dev parquet bucket by @philerooski in #37
• [ETL-385] Grant prod input bucket synapse access by @rxu17 in #38
• [ETL-410] Add namespace for json to parquet and glue database stacks by @rxu17 in #39
• [ETL-378] Launch all stacks to "staging" namespace in production account by @rxu17 in #40
• [ETL-289] Remove deleted and duplicated samples by @philerooski in #41
• [ETL-409] Create python comparison script by @rxu17 in #42
• [ETL-406] Add additional staging resources to production stack group by @rxu17 in #43
• [ETL-414] Add script to generate test S3 event notification by @philerooski in #44
• [ETL-434] Modify S3 to Glue Lambda code to use S3 event notifications by @philerooski in #45
• [ETL-374] Fix failing test for setup_external_storage by @rxu17 in #46

New Contributors

• @rxu17 made their first contribution in #1
• @philerooski made their first contribution in #5
• @thomasyu888 made their first contribution in #26

Full Changelog: https://github.com/Sage-Bionetworks/recover/commits/0.1.0