Add support for InvocationMode.DBT_RUNNER for local execution mode

[jbandoro]

Description

This PR adds dbtRunner programmatic invocation for ExecutionMode.LOCAL. I decided to not make a new execution mode for each (e.g. ExecutionMode.LOCAL_DBT_RUNNER) and all of the child operators but instead added an additional config ExecutionConfig.invocation_mode where InvocationMode.DBT_RUNNER could be specified. This is so that users who are already using local execution mode could use dbt runner and see performance improvements.

With the dbtRunnerResult it makes it easy to know whether the dbt run was successful and logs do not need to be parsed but are still logged in the operator:

Performance Testing

After #827 was added, I modified it slightly to use postgres adapter instead of sqlite because the latest dbt-core support for sqlite is 1.4 when programmatic invocation requires >=1.5.0. I got the following results comparing subprocess to dbt runner for 10 models:

1. InvocationMode.SUBPROCESS:

Ran 10 models in 23.77661895751953 seconds
NUM_MODELS=10
TIME=23.77661895751953

2. InvocationMode.DBT_RUNNER:

Ran 10 models in 8.390100002288818 seconds
NUM_MODELS=10
TIME=8.390100002288818

So using InvocationMode.DBT_RUNNER is almost 3x faster, and can speed up dag runs if there are a lot of models that execute relatively quickly since there seems to be a 1-2s speed up per task.

One thing I found while working on this is that a manifest is stored in the result if you parse a project with the runner, and can be reused in subsequent commands to avoid reparsing. This could be a useful way for caching the manifest if we use dbt runner for dbt ls parsing and could speed up the initial render as well.

I thought at first it would be easy to have this also work for virtualenv execution, since I at first thought the entire execute method was run in the virtualenv, which is not the case since the virtualenv operator creates a virtualenv and then passes the executable path to a subprocess. It may be possible to have this work for virtualenv and would be better suited for a follow-up PR.

Related Issue(s)

closes #717

Breaking Change?

None

Checklist

• I have made corresponding changes to the documentation (if required)
• I have added tests that prove my fix is effective or that my feature works - added unit tests and integration tests.

[netlify]

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name	Link
🔨 Latest commit	ce2efd9
🔍 Latest deploy log	https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/65de1f5888b8d20008f37d86

[jbandoro]

The performance tests for this branch (push) are succeeding, but the performance tests for (pull_request_target) are failing because I added in a check to the peformance dag test to ensure it was successful and the main GH action file does not have the necessary postgres service that was added in this PR.

[tatiana]

This is a fantastic performance improvement, @jbandoro ! Great work!

[jbandoro]

The performance tests are now failing because I removed this branch from the on-push from the test workflow. The performance tests passed in the previous commit here: https://github.com/astronomer/astronomer-cosmos/actions/runs/8068823803