Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CT-1747] Enable flagging nodes/configs for deferred validation #6511

Open
gshank opened this issue Jan 4, 2023 · 1 comment
Open

[CT-1747] Enable flagging nodes/configs for deferred validation #6511

gshank opened this issue Jan 4, 2023 · 1 comment
Labels
tech_debt Behind-the-scenes changes, with little direct impact on end-user functionality

Comments

@gshank
Copy link
Contributor

gshank commented Jan 4, 2023

Currently we have an issue with snapshot config, where the validation requires that certain config attributes be set. Since all node configs are validated twice -- once at model parsing time and once at schema parsing time (if a schema config exists) -- if somebody attempts to set the snapshot config in a schema file it will fail on the model parsing step.

In a more recent initiative we have config that might be set in both/either model and schema file and are attempting to validate that all of the config is in place. If the feature is turned on in model config and the other config settings are in the schema file, the complete validation would fail on model parsing.

What we need to handle situations like this is to be able to flag certain validation situations to happen after all of the schema parsing has been done. For example if a snapshot node is constructed from a sql file but doesn't contain some of the required config, set a flag. If a schema file runs that validation and it succeeds, we can then turn the flag off. But if we reach the end of schema file parsing and the flag is still set, re-run validation and issue an error.

@github-actions github-actions bot changed the title Enable flagging nodes/configs for deferred validation [CT-1747] Enable flagging nodes/configs for deferred validation Jan 4, 2023
@jtcohen6 jtcohen6 added tech_debt Behind-the-scenes changes, with little direct impact on end-user functionality Team:Language labels Jan 4, 2023
@jtcohen6
Copy link
Contributor

Specific case we're interested in:

  • Configure that a model must have constraints_enabled (likely renaming this to contracted: true)
  • Define that config anywhere (dbt_project.yml, yaml file, within .sql file)
  • That config presumes all columns are defined with name + data_type. If any is missing (i.e. data_type), raise a validation error during parsing.

We can fall back to doing this not at parse time, since we'll catch it at runtime anyway. It's conceivable that we'd want this capability for other cases in the future, where we want to validate compatibility between configs/attributes that are being set in different places.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
tech_debt Behind-the-scenes changes, with little direct impact on end-user functionality
Projects
None yet
Development

No branches or pull requests

2 participants