Pass config values to sources in dbt_project.yml

[azhard]

Describe the feature

In the dbt_project.yml file you can pass specific config values to models based on the folder structure. This is extremely useful so that you are not forced to pass in the same value in individual file configs. This pattern however doesn't work for sources. A project.yml sectoin that I would expect to work is as follows:

name: 'dbt'
version: '1.0.0'

config-version: 2

...

# This works
models:
  dbt:
    dir1:
      +tags:
        - fancy_tag1
      sub_dir1:
        +enabled: false
      sub_dir2:
        +materialized: ephemeral

    dir2:
      +tags:
        - fancy_tag2

# This doesn't cause an error but doesn't actually works
sources:
  dbt:
    dir1:
      +database:
        - database_1
      sub_dir1:
        +loaded_at_field: col_1
      sub_dir2:
        +description: Sub_dir2 sources live here

    dir2:
      +loader:
        - Kafka

Additional context

This should work across all databases

Who will this benefit?

Presumably any dbt users who have sources in separate files but those sources share config elements.

[jtcohen6]

Really good points @azhard, and I totally agree. Check out #2401.

There are two kinds of dbt config today:

• prescriptive node configs, set in dbt_project.yml or using the {{ config() }} macro within model files
• descriptive resource configs, a.k.a. the schema.yml spec, set in any /**.yml files that are not dbt_project.yml

Right now, source-specific configs such as database and loader can only be set in resource.yml files. The only possible config for sources in dbt_project.yml today is enabled (docs):

sources:
  my_dir:
    +enabled: true|false

We have a plan to reconcile node configs and resource configs—it's going to be a lot of hard work, and very much on the roadmap.

[azhard]

Thanks @jtcohen6, has there been any thoughts on when this feature is slated to be implemented?

[jtcohen6]

@azhard We added #2401 to our 1.0 list, so it's slated for the medium term. I'm going to close this issue as a duplicate in the meantime.

[tom-rb]

Reviving this issue, I still have the same problem as the OP. Meaning, this doesn't work:

sources:
  project:
    dir1:
      +database: database_1

Was this issue correctly closed then?

[dbeatty10]

Second time's the charm?

Per #9990 (comment), see #7157 for an issue that looks like it will resolve this.