[CT-3438] UserConfig in profiles.yml -> flags in dbt_project.yml

[jtcohen6]

Housekeeping

• I am a maintainer of dbt-core

Short description

We are deprecating config as a thing that is settable in profiles.yml. Instead, we will support flags in dbt_project.yml

For dbt-core v1.8, we should see if the user has defined config in profiles.yml.

• If they have, continue to respect it, and raise a deprecation warning.
• If they haven't, great. Instead, we will be reading flags from dbt_project.yml. The two are mutually exclusive.

In a future release, we will remove UserConfig from profiles.yml entirely.

Acceptance criteria

In dbt_project.yml, it will look like:

# dbt_project.yml
flags:
  send_anonymous_usage_stats: True

Going forward, some flags will be settable in:

1. "Project flags" (dbt_project.yml) only. These flags define behaviors tied to the source code in a project, and should remain consistent across all users, environments, and invocations of that project.
2. CLI option / env var only. These flags are highly specific to the running invocation, and don't make sense to define in version-controlled code. Several of these are file system paths, such as --project-dir or --log-path. Some want to vary across commands, such as --full-refresh. Environment variables are best suited to flags that want different settings in different environments, such as turning DBT_FAIL_FAST on in dev/CI but not in prod.
3. Both places. In this case, a project wants to set a default behavior, but it may be appropriate to override it for different environments/invocations.

If set in both places, the inheritance hierarchy remains the same:

1. CLI var (handled by click)
2. Env var (handled by click)
3. User config Project flags

Implementation Notes

• We should rename the internal UserConfig to ProjectFlags
• User config is read when Flags is instantiated here. This is probably still the right place to do it, but the logic now becomes:
  • read_project_flags:
    • tries to load a user config from profiles.yml. If found, read it and raise a deprecation warning
    • if not found, tries to load user config from dbt_project.yml (can be a minimal yaml parse for just the 'flags' key - no jinja parsing necessary)
    • otherwise, use default user config settings (Nones are appropriate to ensure flag precedence works as expected for flags that can be set via CLI or env var)
• this will also need to be updated in core.dbt.flags.set_from_args

Testing:

• let's add a unit test that ensures that: every attribute of ProjectFlags that has a corresponding click option in params.py should be set to None by default (except for anon user tracking). Going forward, flags can have non-None defaults if they do not have a corresponding CLI option/env var. These will be used to control backwards incompatible interface or behaviour changes.

Impact to Other Teams

N/A

Will backports be required?

no :)

Context

Implications of deprecation

Why was this called UserConfig before? Well... hypothetically, if Michelle wants colorized logs, and Jeremy doesn't, they should be able to set user-specific preferences that apply across all their projects, without checking those preferences into those projects' version control (because profiles.yml lives in the HOME directory on each machine). While we are removing UserConfig from profiles.yml, it will remain possible to set user-specific preferences by setting environment variables (in .bashrc, .zshrc, etc).

[jtcohen6]

From chatting with @gshank: Let's see what the complexity will be for implementation a "soft" deprecation. If it's a lot of additional work, we should instead opt for a "hard" deprecation in v1.8 (with a lot of prior notice), i.e. stop reading config from profiles.yml entirely.

[vianmora]

Hello :)
I'm comming from this Issue : #6207 (comment)
and before, this one : #8026

I'm trying to understand, are you going to allow the use of .env (or dbt.env) files? Or is the project abandoned as the issue is closed ? :)

[jtcohen6]

Hey @vianmora! Thanks for following up here. We kicked that part out of scope for now, to refocus on:

• moving flags (f.k.a. "user configs") from profiles.yml → dbt_project.yml
• defining the intended use of flags for managing behavior changes

A few questions for you:

• Are you still interested in using .env files to set environment variables?
• Are these env vars for global/runtime configs (things like DBT_FAIL_FAST), or for env vars that modify behavior in your project (accessed via {{ env_var(...) }})?
• How are you developing/deploying dbt? (using dbt Cloud or otherwise)

[nikohunt]

Hey @vianmora! Thanks for following up here. We kicked that part out of scope for now, to refocus on:

• moving flags (f.k.a. "user configs") from profiles.yml → dbt_project.yml
• defining the intended use of flags for managing behavior changes

A few questions for you:

• Are you still interested in using .env files to set environment variables?
• Are these env vars for global/runtime configs (things like DBT_FAIL_FAST), or for env vars that modify behavior in your project (accessed via {{ env_var(...) }})?
• How are you developing/deploying dbt? (using dbt Cloud or otherwise)

@jtcohen6
I'm still interested in better handling of .env., please.

• The env vars are for holding profile config (e.g., host, username etc.) and so not checked in to a repo.
• I'm using dbt Core, orchestrated by Dagster.

[conorpp]

It's be great to support loading variables from .env as this is more consistent with other projects (e.g. meltano, dagster). If we can put secrets in .env then we don't need to also duplicate them to dbt's profiles.yml.

[nikohunt]

@conorpp, I've found that adding the prefix DBT_ or DBT_ENV_SECRET_ to my variables in .env seem to cause these .env files to be used by dbt, negating the need for using set -a; source .env; set +a.

[elyobo]

Would this possibly have caused this issue in sqlfluff (that has definitely been caused by something in 1.7.14)? sqlfluff/sqlfluff#5841

[martanthony]

For dbt-core v1.8, we should see if the user has defined config in profiles.yml.

...

Will backports be required?

no :)

Curious to understand why this was also released to 1.6 and 1.7? There's nothing on this ticket or the pull request to indicate this should have affected versions other than 1.8

[jtcohen6]

@elyobo Thanks for flagging - I've responded on the linked sqlfluff issue.

@martanthony You're right, it wasn't our initial intent to backport this change. We decided to do this recently in order to provide existing users with the deprecation warning and mechanism to opt into more-secure (less-surprising) behavior around how built-in materializations from packages are overridden:

• Raise deprecation warning if installed package overrides built-in materialization #9981
• https://docs.getdbt.com/reference/global-configs/legacy-behaviors#package-override-for-built-in-materialization

Apologies for not reflecting that on this issue in advance. Has it caused any issues for you, besides the one noted above for sqlfluff?

[martanthony]

@jtcohen6 understood – thanks for the prompt response. We have a step in our CI that consumes the output of dbt ls and the deprecation warning appears to have broken the way that output is handled by us. Easily mitgated by moving the profile configs to dbt_project flags though.

[polmonso]

Doesn't this change cause that everytime we start a new dbt project it will have telemetry enabled by default if we forget to turn it off?

Is the environment variable DBT_SEND_ANONYMOUS_USAGE_STATS=False the way to go for globally disabling telemetry?

[bmarcj]

@conorpp, I've found that adding the prefix DBT_ or DBT_ENV_SECRET_ to my variables in .env seem to cause these .env files to be used by dbt, negating the need for using set -a; source .env; set +a.

This comment suggests some sort of support for '.env' files now exists in dbt. Is this actually the case (given the lack of documentation and further comment here)?

[elyobo]

@bmarcj I don't think it's built in, but I use direnv to set the variables and then you can reference them in various places in dbt.

[bmarcj]

Thanks @elyobo! I'm looking for a minimally invasive solution for my team, so a shell extension might be out the question.

[elyobo]

I couldn't imagine life without direnv anyway, I use it for all projects. The layout options make things so much less painful as well as the env vars (no more npx <command> or poetry run dbt run or whatever). Makes me wonder why apps bother including support for e.g. .env files at all any more :D

[BI-MarcB]

We currently use

config:
  log_format: json

in the profiles.yml of your production dbt user(s). This way, our devs can enjoy text-logs, while we get json output for Airflow and Datadog. Now that flags in the profile are deprecated, I am struggling to re-implement such behaviour. If I add

flags:
  log_format: "json"

to dbt_project.yml, the console output for devs is also in json. Using jinja in the flags section does not seem to be supported. My numerous tries of variation of the following did not work
log_format: '{% if target.name == "local_dev" %}json{% else %}json{% endif %}'
The above always defaults to text, even if I put json as both parts of the if-else-construct.

How can I display logs in dev console in text, but on production in json? The only "solution" that I can think of is manually adjusting every single of our 50+ dbt calls in the Airflow DAGs by adding "--log-format json" which is cumbersome, error-prone and harder to maintain.