Fix snapshot to always drop the staging table #96
Conversation
| @@ -0,0 +1,16 @@ | |||
| {% macro statement_with_staging_table(name=None, staging_table=None, fetch_result=False, auto_begin=True) -%} | |||
There was a problem hiding this comment.
Maybe we can add a comment here (clean up staging table generated by dbt snapshot)
It is supposed to be deleted at https://github.com/databricks/dbt-databricks/pull/96/files#diff-c139bdc07e2116a9e3beb0cf3110fb25dc2b8f0f36b907595c83f2944ee631eaL105-L107, but if the query fails, it's unreachable. |
| @@ -102,10 +106,6 @@ | |||
|
|
|||
| {{ adapter.commit() }} | |||
|
|
|||
| {% if staging_table is defined %} | |||
| {% do post_snapshot(staging_table) %} | |||
There was a problem hiding this comment.
The default implementation of post_snapshot seems to be a noop
https://github.com/dbt-labs/dbt-core/blob/e7218d3e99837f0139fb7ecd367d3bdf1135a961/core/dbt/include/global_project/macros/materializations/snapshots/helpers.sql#L17-L23
And dbt-spark doesn't seem to implement it either. Where is the logic of post_snapshot defined?
There was a problem hiding this comment.
dbt-spark does have the implementation.
There was a problem hiding this comment.
👍 This doesn't show up in Github's search result..
|
Thanks! merging. |
### Description We introduced Delta constraints at databricks#71 and now that snapshot query could fail when it's violating the constraints. In that case, the temporary view keeps existing because snapshot uses a permanent view and drop it later. We should always drop them.
Description
We introduced Delta constraints at #71 and now that snapshot query could fail when it's violating the constraints.
In that case, the temporary view keeps existing because snapshot uses a permanent view and drop it later.
We should always drop them.