Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ADAP-1024] [Feature] Save tags to Snowflake using object tagging feature #827

Open
3 tasks done
kamilamarcinekpgs opened this issue Nov 9, 2023 · 1 comment
Open
3 tasks done
Labels
enhancement New feature or request

Comments

@kamilamarcinekpgs
Copy link

kamilamarcinekpgs commented Nov 9, 2023

Is this your first time submitting a feature request?

  • I have read the expectations for open source contributors
  • I have searched the existing issues, and I could not find an existing issue for this feature
  • I am requesting a straightforward extension of existing dbt-snowflake functionality, rather than a Big Idea better suited to a discussion

Describe the feature

Besides dbt tags there should be possibility to define 'snowflake-tags' for tables and columns (and possibly schemas), which would be persisted to Snowflake as tags (https://docs.snowflake.com/en/user-guide/object-tagging).

Describe alternatives you've considered

We are aware of the dbt-snowflake-utils package by Montreal Analytics, which now has a macro that creates Snowflake tags from model/column meta.

It is not usable for tag-based masking though, because there is a time gap between materializing a table and creating a tag (currently it creates tags in on-run-end, so when all models are ready), which would create a serious vulnerability in PII protection. Therefore tags need to be a part of the CTAS (same as masking policies problem described in dbt-labs/dbt-adapters#85

Who will this benefit?

Tags are useful for monitoring, but also they enable a powerful feature called tag-based masking, which is getting more and more popular.

Are you interested in contributing this feature?

No response

Anything else?

This feature has already been requested (#104), but closed with suggestion, that the dbt-snowflake-utils macro is a sufficient alternative. It may be for some cases, but we take our PII protection too seriously to enable a time gap between table materialization and tag creation.

@kamilamarcinekpgs kamilamarcinekpgs added enhancement New feature or request triage labels Nov 9, 2023
@github-actions github-actions bot changed the title [Feature] Save tags to Snowflake using object tagging feature [ADAP-1024] [Feature] Save tags to Snowflake using object tagging feature Nov 9, 2023
@michael-the1
Copy link

This is also important for us, for exactly the same reasons as yours. We currently solve this using a custom materialization, but that comes with its own downsides of course.

Implementation is pretty straightforward. For the definition, I like the names dbt-snowflake-utils, though I'd put them at the top level:

# schema.yml

models:
  - name: ACCOUNT
    +schema: FINANCE
    database_tags:
      accounting_row_string: a

    columns:
      - name: ACCOUNT_NAME
        database_tags:
          accounting_col_string: b

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants