Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor partial parsing cache to be per dbt project #1130

Open
tatiana opened this issue Jul 31, 2024 · 0 comments
Open

Refactor partial parsing cache to be per dbt project #1130

tatiana opened this issue Jul 31, 2024 · 0 comments
Labels
area:parsing Related to parsing DAG/DBT improvement, issues, or fixes dbt:parse Primarily related to dbt parse command or functionality
Milestone

Comments

@tatiana
Copy link
Collaborator

tatiana commented Jul 31, 2024

In #904, we started caching the partial parsing and manifest files. However, that happened at a DbtDag and DbtTaskGroup level. By then, we considered configuring at the dbt project level, but there were concerns regarding concurrent tasks running in the same Airflow worker node.

This ticket aims to refactor the caching to be set at a dbt project level. We should consider using the _safe_copy method created in #1086. We should create alphas and have as many users as possible trying out this.

This refactor will improve with filling up users' disk: #1042

@tatiana tatiana added this to the Cosmos 1.7.0 milestone Jul 31, 2024
@dosubot dosubot bot added area:parsing Related to parsing DAG/DBT improvement, issues, or fixes dbt:parse Primarily related to dbt parse command or functionality labels Jul 31, 2024
@tatiana tatiana modified the milestones: Cosmos 1.7.0, Triage Sep 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:parsing Related to parsing DAG/DBT improvement, issues, or fixes dbt:parse Primarily related to dbt parse command or functionality
Projects
None yet
Development

No branches or pull requests

1 participant