Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Insights DAG to load into transitional table before transformed data #190

Open
joshgamache opened this issue Jan 30, 2023 · 0 comments

Comments

@joshgamache
Copy link
Contributor

Description

The initial work done to ingest Telus Insights data into the database was done pulling the data directly into the Data Science Workspace (DSW) segment of the software. To conform with our architecture plan, it should first be pulled into the Data Clean Room, then transitioned into the DSW from there.

User Story

As a developer

I want Telus Insights data to load into the DCR first

So that I can preform PII analysis reporting/anonymization prior to it going into the DSW for general analysis and reporting

Acceptance Criteria

Given When Then Pass/Fail (TBD by reviewer)
I am connected to the database I query for Insights data in the DCR data should return
I am connected to Airflow/Composer I check the Insights ingestion DAG data should flow first into the DCR then into the DSW from there

Developer notes

  • The process flow for this already exists in the Waste and Building datasets, use those as a reference.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
No open projects
Status: Ready for PO Review
Development

No branches or pull requests

1 participant