New global-ish configs: TARGET_PATH + LOG_PATH

[jtcohen6]

Contributions

• I have read the contribution docs, and understand what's expected of me.

Link to the page on docs.getdbt.com requiring updates

• https://docs.getdbt.com/reference/global-configs
• https://docs.getdbt.com/reference/dbt_project.yml
• https://docs.getdbt.com/reference/project-configs/log-path
• https://docs.getdbt.com/reference/project-configs/target-path

What part(s) of the page would you like to see updated?

dbt-core v1.2 adds support for overriding the log-path and target-path set in dbt_project.yml, via:

• CLI flag: --log-path, --target-path
• Env var: DBT_LOG_PATH, DBT_TARGET_PATH

For both types of global/runtime configs, the precedence order is:

1. CLI flag (dbt run --log-path=other_logs)
2. Env var (DBT_LOG_PATH=other_logs dbt run)
3. Yaml file config (dbt_project.yml)

In older versions of dbt, option 2 was already achievable by configuring it in the yaml file like:

# dbt_project.yml
log-path: "{{ env_var('DBT_LOG_PATH', 'logs') }}"

Going forward, that behavior will be implicitly true for all projects.

Additional information

dbt-labs/dbt-core#5402

These are "global [project] configs," subtly different from our existing "global [user/profile] configs." The difference is that log-path and target-path are traditionally defined in dbt_project.yml (per project), instead of the config: block in profiles.yml (per machine).

The source of that tradition is, dbt_project.yml is always in the root directory from which dbt is run, so runtime paths (logs, target, etc) are relative to the project file. Whereas profiles.yml is traditionally in a hidden folder on the machine, outside the root directory / version-controlled file system. (We are thinking about changing that: dbt-labs/dbt-core#5411, dbt-labs/dbt-core#5425)

Unfortunately, it looks like the implementation added support for these flags in a slightly different position from the standard "global config" location. I would've expected dbt --log-path=another run, but it's actually dbt run --log-path=another. Oh well 🤷 we can fix it in post, to make it work in both positions.

(One more reason that, with our work around splitting out + refactoring the dbt-core CLI, we should look to make position matter not at all, and accept flags/arguments wherever they're valid.)