Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cleanup unnecessary codes by the inheritance of dbt-spark any more. #40

Merged
merged 3 commits into from
Feb 4, 2022

Conversation

ueshin
Copy link
Collaborator

@ueshin ueshin commented Jan 28, 2022

Description

Cleans up codes and macros that are unnecessary after the inheritance of dbt-spark any more.

@ueshin ueshin marked this pull request as draft January 28, 2022 23:11
@ueshin ueshin marked this pull request as ready for review January 29, 2022 02:01
Copy link
Contributor

@jtcohen6 jtcohen6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very exciting to see this PR!

{% endif %}
{% endmacro %}


{% materialization snapshot, adapter='databricks' %}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like you discovered this yourselves, but just for the sake of stating clearly:

Materializations don't natively support the same sort of "parent" adapter inheritance as other macros. I'm going to open an issue in dbt-core that addresses this. In the meantime, you can "mock" that inheritance (and avoid duplicated code) via:

{% materialization snapshot, adapter='databricks' %}
    {# this actually calls the spark version of the snapshot materialization #}
    {% set relations = materialization_snapshot_spark() %}
    {{ return(relations) }}
{% endmaterialization %}

This isn't a capability we document or recommend for end users, but we use it in dbt-snowflake as well. Any better approach we implement in a new dbt-core minor version will aim to include a non-breaking upgrade path / backwards compatibility for this syntax.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for letting me know. That's great to know.
Actually for this materialization, there are some updates for dbt-databricks from dbt-spark, so I think we still need to redefine here.

@ueshin
Copy link
Collaborator Author

ueshin commented Feb 4, 2022

Thanks! merging.

@ueshin ueshin merged commit 6402e71 into databricks:main Feb 4, 2022
@ueshin ueshin deleted the cleanup branch February 4, 2022 18:58
ueshin added a commit that referenced this pull request Feb 4, 2022
)

### Description

Cleans up codes and macros that are unnecessary after the inheritance of `dbt-spark` any more.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants