You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently we have an issue with snapshot config, where the validation requires that certain config attributes be set. Since all node configs are validated twice -- once at model parsing time and once at schema parsing time (if a schema config exists) -- if somebody attempts to set the snapshot config in a schema file it will fail on the model parsing step.
In a more recent initiative we have config that might be set in both/either model and schema file and are attempting to validate that all of the config is in place. If the feature is turned on in model config and the other config settings are in the schema file, the complete validation would fail on model parsing.
What we need to handle situations like this is to be able to flag certain validation situations to happen after all of the schema parsing has been done. For example if a snapshot node is constructed from a sql file but doesn't contain some of the required config, set a flag. If a schema file runs that validation and it succeeds, we can then turn the flag off. But if we reach the end of schema file parsing and the flag is still set, re-run validation and issue an error.
The text was updated successfully, but these errors were encountered:
github-actionsbot
changed the title
Enable flagging nodes/configs for deferred validation
[CT-1747] Enable flagging nodes/configs for deferred validation
Jan 4, 2023
Configure that a model must have constraints_enabled (likely renaming this to contracted: true)
Define that config anywhere (dbt_project.yml, yaml file, within .sql file)
That config presumes all columns are defined with name + data_type. If any is missing (i.e. data_type), raise a validation error during parsing.
We can fall back to doing this not at parse time, since we'll catch it at runtime anyway. It's conceivable that we'd want this capability for other cases in the future, where we want to validate compatibility between configs/attributes that are being set in different places.
Currently we have an issue with snapshot config, where the validation requires that certain config attributes be set. Since all node configs are validated twice -- once at model parsing time and once at schema parsing time (if a schema config exists) -- if somebody attempts to set the snapshot config in a schema file it will fail on the model parsing step.
In a more recent initiative we have config that might be set in both/either model and schema file and are attempting to validate that all of the config is in place. If the feature is turned on in model config and the other config settings are in the schema file, the complete validation would fail on model parsing.
What we need to handle situations like this is to be able to flag certain validation situations to happen after all of the schema parsing has been done. For example if a snapshot node is constructed from a sql file but doesn't contain some of the required config, set a flag. If a schema file runs that validation and it succeeds, we can then turn the flag off. But if we reach the end of schema file parsing and the flag is still set, re-run validation and issue an error.
The text was updated successfully, but these errors were encountered: