[CT-2430] [Freshness test tags] Please add tags to freshness tests

[mddataminr]

Is this your first time submitting a feature request?

• I have read the expectations for open source contributors
• I have searched the existing issues, and I could not find an existing issue for this feature
• I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

I would like Freshness tests to be tagged and then allow running specific tags using some sort of a select query. This would allow for more granular running of tests.

Describe alternatives you've considered

Discontinue using Freshness tests and instead write a custom macro

Who will this benefit?

Potential benefits:

• Tests can be grouped by SLA priority, allowing for better on call alert handling
• This would save us compute costs because if we have a long failure period, we don't need to run the tests as often as others
• Would speed up development cycles if we can run only tests related to the changes in the feature being implemented

Are you interested in contributing this feature?

I'm new to the code in this project. If someone can point me in the right direction I can make the changes

Anything else?

No response

[dbeatty10]

Thanks for proposing this @mddataminr !

Are you thinking something like this?

models/_sources.yml

version: 2

sources:
  - name: dbt_dougie_fresh
    config:
      tags: ["better_be_fresh"]
    freshness:
      warn_after: {count: 12, period: hour}
      error_after: {count: 24, period: hour}
    loaded_at_field: _etl_loaded_at
    database: postgres
    schema: dbt_dougie_fresh
    tables:
      - name: my_model

Then only check that source (but none others) via its tag like this?

dbt source freshness --select tag:better_be_fresh   

[mddataminr]

I wasn't thinking the tags would be in the config. I was thinking it'd be on sources, but I'm good with it in config. Having it in config might open things up to add more in the future. In general, that's exactly the functionality I'd be looking for!

…

On Tue, Apr 18, 2023, 8:47 PM Doug Beatty ***@***.***> wrote: Thanks for proposing this @mddataminr <https://github.com/mddataminr> ! Are you thinking something like this? models/_sources.yml version: 2 sources: - name: dbt_dougie_fresh config: tags: ["better_be_fresh"] freshness: warn_after: {count: 12, period: hour} error_after: {count: 24, period: hour} loaded_at_field: _etl_loaded_at database: postgres schema: dbt_dougie_fresh tables: - name: my_model Then only check that source (but none others) via its tag like this? dbt source freshness --select tag:better_be_fresh — Reply to this email directly, view it on GitHub <#7395 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AXNEDBR6R2JBZRPQRVLU5Y3XB472VANCNFSM6AAAAAAXCXXICM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- This message (and any associated files) is intended only for the use of the individual or entity to which it is addressed and may contain information that is confidential and/or proprietary. If you are not the intended recipient, you are hereby notified that any dissemination, copying or distribution of this message, or files associated with this message, is strictly prohibited. If you have received this message in error, please notify us immediately by replying to the message and deleting it.

[jtcohen6]

@dbeatty10 brilliant example - this actually works if I just defined the tags outside the config dictionary:

version: 2

sources:
  - name: dbt_dougie_fresh
    tags: ["better_be_fresh"]
    freshness:
      warn_after: {count: 12, period: hour}
      error_after: {count: 24, period: hour}
    loaded_at_field: _etl_loaded_at
    database: postgres
    schema: dbt_dougie_fresh
    tables:
      - name: mixtapes

$ dbt source freshness --select tag:better_be_fresh
05:35:50  Running with dbt=1.4.5
...
05:35:51  1 of 1 START freshness of dbt_dougie_fresh.mixtapes ............................ [RUN]

I know that's not totally consistent with how this works for other resource types. As a pleasant surprise, we do actually document that fact (https://docs.getdbt.com/reference/resource-configs/tags#other-resource-types):

Tags can also be applied to sources, exposures, and even specific columns in a resource. These resources do not yet support the config property, so you'll need to specify the tags as a top-level key instead.

---

We could reformulate this into a separate feature request:

• Now that sources support that config property (since v1.1), support tags nested under the config dictionary, instead of at the top level, for consistency with other resource types

We've already got some spike work planned over the next few months, to take another good look at the discrepancies between configs and properties, and opportunities for consolidation:

• [CT-2296] [Spike] Remove the distinction between configs and non-configs #7157

So I'm going to close this issue as resolved for the time being.