Set global variable overrides on the command line with --vars

[drewbanin]

What is this?

Variables are a useful way to dynamically configure models and macros. This PR makes these already powerful variables even more flexible by permitting variable declarations on the command line. Now, dbt users can alter the semantics of their dbt runs without changing any code.

Usage

$ dbt run --vars '{"key": "value"}'

Details

The --vars argument accepts a YAML dictionary as a string on the command line. YAML is convenient here, because it's a superset of JSON. As such, arbitrarily nested data structures can be provided to dbt as variables using this syntax.

In the majority of cases, variables are not arbitrarily nested. Instead, they're simple key-value pairs. In these scenarios, YAML is convenient because it does not require strict quoting as with JSON.

Both of the following are all valid and equivalent:

$ dbt run --vars '{"key": "value", "date": 20180101}'
$ dbt run --vars '{key: value, date: 20180101}'

If only one variable is being set, the brackets are optional, eg:

$ dbt run --vars 'key: value'

There's a useful doc on YAML semantics here

Precedence

These variables are applied at the "root" level, as though they were defined directly under models: in the top-level dbt_project.yml file. Variables defined here will take precedence over any variable declarations defined anywhere else in your project or its dependencies.

The order of precedence for variable declaration is now:

1. The variable's default argument (if one is provided)
2. The most specific variable definition in the model's own dbt_project.yml file
3. The most specific variable definition in the root dbt_project.yml file
4. The CLI --vars definition

Notes:

• Item 2 above is only relevant if the model is inside of a package.
• If vars are defined inside of a macro, precedence is determined based on the model(s) which call that macro

[cmcarthur]

this code looks fine. but for some reason I thought there was a builtin argparse yaml parser that pushed the syntax in this direction. am I misremembering that?

[drewbanin]

@cmcarthur not that i know of!

[cmcarthur]

ok,