[CT-299] Decouple project configs from Task logic

[iknox-fa]

Is there an existing feature request for this?

• I have searched the existing issues

Describe the Feature

Generated from #4630

User Story:

In order to access project data earlier in the runtime process dbt needs to be able to generate a ProjectConfig object before starting the task.

TL;DR

I can envision a few different ways to meet his need with varying degrees of ease and correctness

• Method 1: Duplicate all the logic needed to generate project configs and isolate it without any of the Task related specificity.
  • Pros: Fast and easy-ish.
  • Cons: Meh at best, not DRY, not easy to maintain, adds more complexity.
• Method 2: Thread together the various class methods needed to generate the project config. Only the basic flow code
  • Pros: Almost as easy as option 1, Less duplicated code than option 1.
  • Cons: Still messy and confusing, doesn't address cli/base task mess.
• Method 3: Create the project config (and the profile config) at the outset, then pass them into the selected Task. will require doing quite a bit of fussing with the cli and Task code.
  • Pros: Would greatly improve our Task orchestration code's readability and simplicity, Would make testing much easier.
  • Cons: Not easy

Describe alternatives you've considered

No response

Who will this benefit?

No response

Are you interested in contributing this feature?

No response

Anything else?

Research:

Task dependency graph

Notes on Pre-task configs, tracing through the codebase to generation

Arguments

From command line

• main.py:
  • sys.argv ->
  • main() ->
  • handle_and_check() ->
  • parse_args() ->
    • Config: parsed_args <class 'argparse.Namespace'>

Flags

Partially depends on parsed_args @ core/dbt/flags.py:102

• main.py:
• flags.py: (imported)
  • Config: flags (global module w module constants)

Contents of user profile

Read multiple times, this is the first one
Depends on flags

• main.py:
  • handle_and_check() ->
• config/profile.py:
  • read_user_config() ->
    • Config: user_config <class 'dbt.contracts.project.UserConfig'>

The three configs above are passed into each task. They also determine which task is called
@ core/dbt/main.py:214

Task configs

Runtime / UnsetProfile configs

• task/base.py:
  • ConfiguredTask.from_args() ->
  • move_to_nearest_project_dir() -> (dir structure and CWD as config?!)
  • BaseTask.from_args() ->
• core/dbt/task/any_given_class.py:
  • ConfigType.from_args() ->
• core/dbt/config/runtime.py:
  • RuntimeConfig.from_args() ->
  • RuntimeConfig.collect_parts() ->
    • Config: project <class 'dbt.config.project.Project'>
    • Config: profile <class 'dbt.config.profile.Profile'>
  • RuntimeConfig.from_parts() ->
    • Config: retval <class 'dbt.config.runtime.RuntimeConfig'>
      OR
  • UnsetProfileConfig.from_args() ->
  • RuntimeConfig.collect_parts() ->
    • Config: project <class 'dbt.config.project.Project'>
    • Config: profile <class 'dbt.config.profile.Profile'>
  • UnsetProfileConfig.from_args() ->
    • Config: retval <class 'dbt.config.runtime.UnsetProfileConfig'>

Tasks require either RuntimeConfig or UnsetProfileConfig, both of which are created from a
Project, Profile, and args (excepting a few that don't require config data at all, see diagram)