Fix equality test with ephemeral model + explicit column set #321
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See: dbt-labs/spark-utils#7
This is a:
master
dev/
branchdev/
branchDescription & motivation
If
equality
is comparing an ephemeral model with thecompare_columns
keyword argument, we want to avoid runningadapter.get_columns_in_relation(model)
entirely.Currently, it is the default/fallback arg to
kwargs.get()
. Instead, we should place it inside a conditional that only runs ifkwargs.get(compare_columns)
returns none. I was under the impression that these were effectively the same, but this change actually fixes an integration test (test_equal_column_subset
) ondbt-utils
+spark-utils
.Why did this only break with Spark? Because
dbt-spark
usesdescribe table [table_name]
to get the columns from a given table, a command that fails if the table does not exist. Most "core" adapters query the information schema instead; if the table doesn't exist, the query returns no results, but it doesn't fail explicitly.Checklist
I have updated the README.md (if applicable)I have added tests & descriptions to my models (and macros if applicable)