Split out dbt front end for many benefits

[nathaniel-may]

The Problems

• Developing dbt often requires a dbt project that touches a specific code path, and these projects can be cumbersome to create or find.
• Integration test failures are often hard to debug, and converting these tests into local projects is often non-obvious.
• Our test suite is lacking coverage when it comes to reading in dbt projects from a system since that's not how we set up our test projects.
• Creating an new integration test is unnecessarily cumbersome

Action Items

• [] Create dbt projects with the full file structure for local dev, and for the intent of adding test coverage to the part of dbt that reads in system state.
• [] Split out the CLI Loosely Couple CLI #4179
• [] Read environment variables up front (users can define these inside their dbt projects so this might be tricky)
• [] Do all file IO up front
• [] Put all this information in to one datatype that isn't the manifest that can be passed to the next part of dbt to trigger all the flows without needing to do any additional IO.

What this gets us

• Integration test states that can be created by creating or modifying a single Python value.
• Transformation from integration test states to local projects with a simple function.
• Easier to add more coverage for how dbt reads in a project from system state to this internal data type. (These projects could be used for local development since they would already set up properly)
• Concentrates IO logic for better code quality and easier debugging
• A better deserialization target for executing dbt over another interface i.e. client-server

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest; add a comment to notify the maintainers.