You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
MichelleArk
changed the title
[CT-2126] Enforce model contracts for python models
[CT-2126] [Spike] Enforce model contracts for python models
Feb 23, 2023
For SQL models, we run an "empty" version of the model query (where false limit 0) to extract the output schema in advance. I don't think any equivalent exists for Python models.
We could have the option of:
Always create the model in a temporary location
Compare the output result to the contracted set of columns, and raise an error if they differ
Apply constraints after-the-fact, and raise an error if their enforcement fails
IFF the contract enforcement passes, atomically replace the existing table with the current version of the model (whether alter swap, or clone, or create or replace {tbl} as select * from {tmp_tbl})
The biggest downsides are:
(For end users) Contract enforcement will take much more time & compute, versus for SQL models
(For us & adapter maintainers) This would require slightly different & trickier codepaths for contract enforcement, within create_table_as and py_write_table
This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.
Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.
If possible, enforce model contracts for python models.
Otherwise, keep the parsing error we're currently if contracts are used on python models around.
Relevant discussion: https://github.com/dbt-labs/dbt-core/pull/6271/files#r1084049809
The text was updated successfully, but these errors were encountered: