Model level tests to do not run when using -m flag (e.g., relationships and others)

[jgooly]

Describe the bug

Model level tests do not run when using the -m flag, e.g, dbt test -m some_model command. However, model level tests do run when running dbt test command.

Steps To Reproduce

1. Add model level test to any model in testing yml file (e.g., equal_rowcount and relationships for some_model).

From test.yml:

version: 2

models:
  - name: fact_event # some_model
    tests:
      - dbt_utils.equal_rowcount:
          compare_model: ref('stg_fivetran_audit__event')
    columns:
      - name: eventid
        tests:
          - unique
          - not_null
          - relationships:
              to: ref('stg_fivetran_audit__event')
              field: eventid

2. Run dbt test -m some_model.

Expected behavior

Expected 4 tests to run.

Actual results

Only 2 tests executed.

When running dbt test -m fact_event:

dbt test -m fact_event
Running with dbt=0.20.0
Found 73 models, 198 tests, 1 snapshot, 8 analyses, 434 macros, 0 operations, 0 seed files, 53 sources, 0 exposures

11:30:21 | Concurrency: 30 threads (target='dev')
11:30:21 | 
11:30:21 | 1 of 2 START test not_null_fact_event_eventid........................ [RUN]
11:30:21 | 2 of 2 START test unique_fact_event_eventid.......................... [RUN]
11:30:22 | 1 of 2 PASS not_null_fact_event_eventid.............................. [PASS in 1.65s]
11:30:22 | 2 of 2 PASS unique_fact_event_eventid................................ [PASS in 1.71s]
11:30:22 | 
11:30:22 | Finished running 2 tests in 3.50s.

Completed successfully

Done. PASS=2 WARN=0 ERROR=0 SKIP=0 TOTAL=2

Interestingly, when running dbt test or dbt test -m path/to/models, all 4 tests are executed.

When running dbt test -m path/to/models:

dbt test -m path/to/models

Running with dbt=0.20.0
Found 73 models, 128 tests, 1 snapshot, 8 analyses, 434 macros, 0 operations, 0 seed files, 53 sources, 0 exposures

11:50:05 | Concurrency: 30 threads (target='dev')
11:50:05 | 
11:50:05 | 1 of 4 START test dbt_utils_equal_rowcount_fact_event_ref_stg_fivetran_audit__event_ [RUN]
11:50:05 | 2 of 4 START test not_null_fact_event_eventid........................ [RUN]
11:50:05 | 3 of 4 START test relationships_fact_event_eventid__eventid__ref_stg_fivetran_audit__event_ [RUN]
11:50:05 | 4 of 4 START test unique_fact_event_eventid.......................... [RUN]
11:50:08 | 1 of 4 PASS dbt_utils_equal_rowcount_fact_event_ref_stg_fivetran_audit__event_ [PASS in 3.50s]
11:50:08 | 2 of 4 PASS not_null_fact_event_eventid.............................. [PASS in 3.54s]
11:50:13 | 4 of 4 PASS unique_fact_event_eventid................................ [PASS in 8.52s]
11:50:15 | 3 of 4 PASS relationships_fact_event_eventid__eventid__ref_stg_fivetran_audit__event_ [PASS in 10.38s]
11:50:15 | 
11:50:15 | Finished running 4 tests in 12.06s.

Completed successfully

Done. PASS=4 WARN=0 ERROR=0 SKIP=0 TOTAL=4

Screenshots and log output

See above.

System information

Which database are you using dbt with?

• postgres
• redshift
• bigquery
• snowflake
• other (specify: ____________)

The output of dbt --version:

installed version: 0.20.0
latest version: 0.20.0
Plugins:
  - bigquery: 0.20.0
  - snowflake: 0.20.0
  - redshift: 0.20.0
  - postgres: 0.20.0```

The operating system you're using:
macOS Big Sur 11.4

The output of python --version:
Python 3.8.10

Additional context

This bug impacts both the native dbt tests and tests from the dbt_utils package (I have only tested these two).

packages:
  - package: calogica/dbt_date
    version: 0.4.0

dbt deps
Running with dbt=0.20.0
Installing calogica/[email protected]
  Installed from version 0.4.0
  Installing dbt-labs/[email protected]
  Installed from version 0.7.1

[joellabes]

This is intentional behaviour as of dbt 0.20.0 - you can read more about it in the original issue and the PR that resolves it. If you're only going to read one, the PR does a good job of summarising the results of the issue discussion.

I assume that path/to/models contains both the fact_event and stg_fivetran_audit__event models? That's why this works - both nodes are in scope of the -m flag so the relationship tests happen.

I did some quick experiments with the @ operator to try and get the same results you may have expected from prior versions of dbt, but couldn't make it work sorry. There might be a way to do it that I don't know though!

[jtcohen6]

(@joellabes Thanks for linking to the issue and the PR!)

@jgooly Thanks for the detailed write-up above—you're not the only person who found this change confusing in v0.20. I shared some lengthier responses to very similar issues, over in #3496 and #3571. I'd encourage you to give one of those linked issues/PRs a read.

To make a long story short, "indirect" test selection is quite powerful, and quite subtle. I believe the changes in v0.20 help us strike the right balance, preserving the essential intuitive behavior without being too greedy. I'd prefer to err on the side of selecting too few resources rather than too many, and give users finer-grained control over which tests they wish to run.

[jgooly]

Thanks @joellabes and @jtcohen6 - apologies for the duplication. All makes sense and I will utilize the graph operators to achieve the goal.