[Regression] dbt --quiet ls --output json should not print warnings in 1.6 and 1.7

[darist]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

I observe this problem with dbt-core 1.7.14. I do not observe this problem with dbt-core 1.7.11

The following command prints a warning, which makes the output invalid json

dbt --quiet ls --output json

[01:19:50  [WARNING]: Deprecated functionality,,User config should be moved from the 'config' key in profiles.yml to the 'flags' key in dbt_project.yml.,{"name": ...

Expected Behavior

Output should be valid json

dbt --quiet ls --output json

{"name": ...

Steps To Reproduce

profiles.yml file:

foo_profile:
  target: bigquery
  outputs:
    bigquery:
      type: bigquery
      method: "{{ var('method', 'oauth') }}"
      keyfile: "{{ env_var('GOOGLE_APPLICATION_CREDENTIALS', '') }}"
      project: "{{ var('data_project', '<redacted>') }}"
      dataset: "{{ var('dataset', 'dbt_' ~ env_var('USER', 'unknown_user')) }}"
      retries: 2
      threads: 8
      timeout_seconds: 300
      location: US

config:
  send_anonymous_usage_stats: False

Then execute

dbt --quiet ls --output json

With dbt-core 1.7.14:

$ dbt --version
Core:
  - installed: 1.7.14
  - latest:    1.7.14 - Up to date!

Plugins:
  - bigquery: 1.7.7  - Update available!
  - postgres: 1.7.14 - Up to date!

$ dbt --quiet ls --output json
01:30:30  [WARNING]: Deprecated functionality

User config should be moved from the 'config' key in profiles.yml to the 'flags' key in dbt_project.yml.
{"name": "...<redacted>

With dbt-core 1.7.11

$ dbt --quiet ls --output json
{"name": "...<redacted>

Relevant log output

see above

Environment

- OS: Linux
- Python: 3.11.8
- dbt:


$ dbt --version
Core:
  - installed: 1.7.14
  - latest:    1.7.14 - Up to date!

Plugins:
  - bigquery: 1.7.7  - Update available!
  - postgres: 1.7.14 - Up to date!

Which database adapter are you using with dbt?

bigquery

Additional Context

No response

[dbeatty10]

Thanks for reporting this @darist !

Digging into the details

Looks like the change that added the ProjectFlagsMovedDeprecation warning for you was introduced in dbt-core 1.7.14 - May 02, 2024:

• Move flags from UserConfig in profiles.yml to flags in dbt_project.yml #9183 / #9998

This would also likely be an issue in 1.6 and 1.8 because this is included in these PRs as well:

• dbt-core 1.6.14 - May 02, 2024: #10008
• dbt-core 1.8.0-b1 - February 28, 2024: #9289

Things I tried

Attempt 1

Using dbt-core 1.7.14, I tried both of the following, neither of which had any effect:

dbt --warn-error-options '{silence: all}' list -q 
dbt --warn-error-options '{"silence": "ProjectFlagsMovedDeprecation"}' list -q

Log output:

15:16:00  [WARNING]: Deprecated functionality

User config should be moved from the 'config' key in profiles.yml to the 'flags' key in dbt_project.yml.

Attempt 2

Attempting the same commands in dbt-core 1.8.0rc1 raised a ValidationError, and the stack trace ended with:

    raise ValidationError(f"{item} is not a valid dbt error name.")
dbt_common.dataclass_schema.ValidationError: P is not a valid dbt error name.

Attempt 3

I also tried adding this to dbt_project.yml:

flags:
  warn_error_options:
    silence:
      - ProjectFlagsMovedDeprecation

This gave a different error:

Runtime Error
  Do not specify both 'config' in profiles.yml and 'flags' in dbt_project.yml. Using 'config' in profiles.yml is deprecated.

Next steps

In the ideal world for dbt-core 1.6 and 1.7, both errors and warnings should be completely silenced by --quiet without users needing to do anything else. But practically, we might have some other considerations that come into play.

Labeling this as refinement for us to figure out how users should silence warnings like this in versions 1.6, 1.7, and 1.8+.

[yu-iskw]

@dbeatty10 I am wondering how I should handle this with dbt 1.8. Personally, I think it would be good to print out warnings and errors as standard error, not as standard out. By doing so, those doesn't disturb output. Our issue is that the warning message appears in standard out. So, we can't parse resulting output of a dbt Macro with run-operation. What do you think?

[darist]

Trying to upgrade to v1.8.4, but this is still a problem. Any idea on ETA for a fix? Thanks!

[dbeatty10]

@darist we don't have a timeline for this.

But I agree that all warnings should be suppressed whenever the --quiet flag is included.

See below for an example where the UnusedResourceConfigPath warning is suppressed but the ProjectFlagsMovedDeprecation warning isn't.

My (totally unconfirmed) guess is that it has something to do with profiles.yml being parsed earlier than dbt_project.yml and somehow the configuration of the logger is changing in between the parsing of those two files.

Reprex

Create a dbt project that will trigger warnings for both ProjectFlagsMovedDeprecation and UnusedResourceConfigPath:

dbt_project.yml

name: "my_project"
version: "1.0.0"
config-version: 2
profile: "postgres"

models:
  my_project:
    nonexistent_folder:
      +materialized: view

profiles.yml

config:
  use_colors: false

Then list the resources in the project:

dbt list

$ dbt list
00:25:51  [WARNING]: Deprecated functionality

User config should be moved from the 'config' key in profiles.yml to the 'flags' key in dbt_project.yml.
00:25:51  Running with dbt=1.8.3
00:25:51  Registered adapter: postgres=1.8.2
00:25:51  [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
There are 1 unused configuration paths:
- models.my_project.nonexistent_folder
my_project.my_model

Try again with the --quiet flag included and note that the 2nd warning is suppressed from the terminal output whereas the 1st warning still gets through:

$ dbt list --quiet
00:26:46  [WARNING]: Deprecated functionality

User config should be moved from the 'config' key in profiles.yml to the 'flags' key in dbt_project.yml.
my_project.my_model

[dbeatty10]

@yu-iskw I think the only workaround right now is to make the code change to make this warning go away entirely -- in this case, move content from the config: key in profiles.yml to the flags: key in dbt_project.yml.

[dbeatty10]

#10534 will be included in v1.9 when it is released.

#10541 is the backport that will be included in the next patch release for v1.8.