Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Partial parsing updates #1835

Closed
drewbanin opened this issue Oct 15, 2019 · 0 comments · Fixed by #1836
Closed

Partial parsing updates #1835

drewbanin opened this issue Oct 15, 2019 · 0 comments · Fixed by #1836
Labels
enhancement New feature or request partial_parsing

Comments

@drewbanin
Copy link
Contributor

Describe the feature

dbt currently accepts a command line flag, --partial-parse, which enables partial parsing of a dbt project. Two changes are required to make this flag ready for primetime:

  1. The flag should be moved to the config: block in the profiles.yml file
  2. The partial parse .pickle file should record which version of dbt it was produced with, and dbt should not try to re-use a partial_parse.pickle file across versions of dbt.

Moving --partial-parse to the profile

The big idea here is that --partial-parse is cumbersome to apply in repeated invocations of dbt run. By supporting this flag in the config: block of the profiles.yml file, users can opt-in to partial parsing without needing to specify the flag in every invocation of dbt. Partial parsing should remain an opt-in functionality.

dbt may preserve the --partial-parse flag (and optionally support a --no-partial-parse flag) which overrides the behavior defined in the profiles.yml config. This flag is not strictly necessary to support, but may be useful especially in debugging contexts.

Recording dbt versions in the partial_parse.pickle file

The partial_parse.pickle file should be written with a dbt_version: key which provides the dbt version that the pickle file was written with. If the version recorded in the .pickle file does not match the version of dbt that is running, then dbt should not attempt to partial-parse the project.

@drewbanin drewbanin added the enhancement New feature or request label Oct 15, 2019
@drewbanin drewbanin added this to the Louisa May Alcott milestone Oct 15, 2019
beckjake added a commit that referenced this issue Oct 16, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request partial_parsing
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants