Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] Environment variables take precedence over CLI flags that appear after the subcommand #10304

Closed
2 tasks done
joellabes opened this issue Jun 13, 2024 · 8 comments · Fixed by #10423
Closed
2 tasks done
Assignees
Labels

Comments

@joellabes
Copy link
Contributor

joellabes commented Jun 13, 2024

Is this a new bug in dbt-core?

  • I believe this is a new bug in dbt-core
  • I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

As requested by @graciegoheen

The docs say

The most specific setting “wins.” If you set the same flag in all three places, the CLI option will take precedence, followed by the environment variable, and finally, the value in dbt_project.yml. If you set the flag in none of those places, it will use the default value defined within dbt.

But the observed precedence distinguishes between flags before the subcommand vs. after the subcommand

Acceptance criteria

  1. CLI flags take precedence over environment variables regardless if they appear before the subcommand vs. after the subcommand.
  2. These commands give identical output:
    DBT_QUIET=0 dbt --quiet list
    DBT_QUIET=0 dbt list --quiet

Steps To Reproduce

See Doug's comment below

@joellabes joellabes added bug Something isn't working triage labels Jun 13, 2024
@dbeatty10 dbeatty10 self-assigned this Jun 14, 2024
@dbeatty10
Copy link
Contributor

Thanks for reporting this @joellabes 🤩

See below for a simple way to reproduce your report -- even if there's no resources defined in a project.

Reprex

DBT_QUIET=0 dbt --quiet list
DBT_QUIET=0 dbt list --quiet
(dbt_1.8) $ DBT_QUIET=0 dbt --quiet list

(dbt_1.8) $ DBT_QUIET=0 dbt list --quiet

13:44:44  Running with dbt=1.8.1
13:44:44  Registered adapter: duckdb=1.8.1
13:44:44  No nodes selected!

Woah! Different output 🤯

Observed precedence vs. documented precendece

Our documented order of precedence is:

  1. CLI flag
  2. Environment variable
  3. Flags in dbt_project.yml
  4. Default

❌ But the observed precedence distinguishes between flags before the subcommand vs. after the subcommand:

  1. CLI flag before subcommand
  2. Environment variable
  3. CLI flag after subcommand 👈
  4. Flags in dbt_project.yml
  5. Default

Affected versions

The reprex above gives the same result for 1.7 and 1.8. Although this example won't work on 1.6 and 1.5, I'd guess those versions are affected as well.

@dbeatty10
Copy link
Contributor

Acceptance criteria

  1. CLI flags take precedence over environment variables regardless if they appear before the subcommand vs. after the subcommand.
  2. These commands give identical output:
    DBT_QUIET=0 dbt --quiet list
    DBT_QUIET=0 dbt list --quiet

@graciegoheen
Copy link
Contributor

graciegoheen commented Jun 14, 2024

Just to be clear - sounds like it's not actually related to whether or not you set the environment variable inline (i.e. $ SOME_ENV_VAR=value dbt run) but actually about whether you place the flag before or after the subcommand?

@dbeatty10
Copy link
Contributor

Correct @graciegoheen -- I tested out "inline" environment variables vs. exporting environment variables, and they both behaved the same. But when I moved the CLI flag either before or after the subcommand, then I saw the precedence change.

@dbeatty10 dbeatty10 removed their assignment Jun 14, 2024
@dbeatty10 dbeatty10 removed the triage label Jun 14, 2024
@dbeatty10
Copy link
Contributor

dbeatty10 commented Jun 14, 2024

Brief history

The dbt CLI started using the click library starting in v1.5 (replacing argparse). At this time, certain flags needed to be passed before the subcommand and other flags needed to be passed after. Additionally, some flags played musical chairs between 1.4 and 1.5. Confusing for end users! 🤯

Then we introduced support supporting flags in any order, either before or after [1] [2]. This was released in dbt-core 1.7.0 and backported to 1.5 and 1.6. ✨

Finally, we updated our CLI documentation to specify that flags should be passed after subcommands. Passing them before is considered legacy functionality that may be removed in the future [3] [4].

@graciegoheen
Copy link
Contributor

Thanks for that information @dbeatty10 - if this is for legacy behavior, I'm not sure we should spend the effort fixing this

@dbeatty10
Copy link
Contributor

It's actually the opposite!

What we've described as legacy behavior is working as expected. But what we've said that people should do is not working as expected.

@graciegoheen graciegoheen changed the title [Bug] A global config's environment variable can override the same config's CLI flag, even though the flag should win [Bug] CLI flags take precedence over environment variables regardless if they appear before the subcommand vs. after the subcommand. Jun 24, 2024
@graciegoheen
Copy link
Contributor

Updated the ticket accordingly!

@gshank gshank self-assigned this Jul 9, 2024
@dbeatty10 dbeatty10 changed the title [Bug] CLI flags take precedence over environment variables regardless if they appear before the subcommand vs. after the subcommand. [Bug] Environment variables take precedence over CLI flags that appear after the subcommand Jul 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants