Introduce ground work for ExecutionMode.AIRFLOW_ASYNC

[pankajkoti]

This PR is the groundwork for the implementation of ExecutionMode.AIRFLOW_ASYNC (#1120), which - once all other epic tasks are completed - will enable asynchronous execution of dbt resources using Apache Airflow’s deferrable operators. As part of this work, this PR introduces a new option to the enum ExecutionMode : AIRFLOW_ASYNC. When this execution mode is used, Cosmos now creates a setup task that will pre-compile the dbt project SQL and make it available to the remaining dbt tasks. This PR, however, does not yet leverage Airflow's deferrable operators. If users use ExecutionMode.AIRFLOW_ASYNC they will actually be running ExecutionMode.LOCAL operators with this change. The PR (#1230) has a first experimental version of using deferrable operators for task execution.

Setup task as the ground work for a new Execution Mode: ExecutionMode.AIRFLOW_ASYNC:

• Adds a new operator, DbtCompileAirflowAsyncOperator, as a root task(analogous to a setup task) in the DAG, running the dbt compile command and uploading the compiled SQL files to a remote storage location for subsequent tasks that fetch these compiled SQL files from the remote storage and run them asynchronously using Airflow's deferrable operators.

Airflow Configurations:

• remote_target_path: Introduces a configurable path to store dbt-generated files remotely, supporting any storage scheme that works with Airflow’s Object Store (e.g., S3, GCS, Azure Blob).
• remote_target_path_conn_id: Allows specifying a custom connection ID for the remote target path, defaulting to the scheme’s associated Airflow connection if not set.

Example DAG for CI Testing:

Introduces an example DAG (simple_dag_async.py) demonstrating how to use the new execution mode(The execution like mentioned earlier would still run like Execution Mode LOCAL operators at the moment with this PR alone)
This DAG is integrated into the CI pipeline to run integration tests and aims at verifying the functionality of the ExecutionMode.AIRFLOW_ASYNC as and when implementation gets added starting with the experimental implementation in #1230 .

Unit & Integration Tests:

• Adds comprehensive unit and integration tests to ensure correct behavior.
• Tests include validation for successful uploads, error handling for misconfigured remote paths, and scenarios where remote_target_path are not set.

Documentation:

• Adds detailed documentation explaining how to configure and set the ExecutionMode.AIRFLOW_ASYNC.

Scope & Limitations of the feature being introduced:

1. This feature is meant to be released as Experimental and is also marked so in the documentation.
2. Currently, it has been scoped for only dbt models to be executed asynchronously (being worked upon in PR Experimental BQ support to run dbt models with ExecutionMode.AIRFLOW_ASYNC #1230), while other resource types would be run synchronously.
3. BigQuery will be the only supported target database for this execution mode ((being worked upon in PR Experimental BQ support to run dbt models with ExecutionMode.AIRFLOW_ASYNC #1230).

Thus, this PR enhances Cosmos by providing the ground work for more efficient execution of long-running dbt resources

Additional Notes:

• This feature is planned to be introduced in Cosmos v1.7.0.

related: #1134

[netlify]

✅ Deploy Preview for sunny-pastelito-5ecb04 ready!

Name	Link
🔨 Latest commit	481e149
🔍 Latest deploy log	https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/66fb149631707200089ba001
😎 Deploy Preview	https://deploy-preview-1224--sunny-pastelito-5ecb04.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.87%. Comparing base (e0a9fd3) to head (3565723).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1224      +/-   ##
==========================================
+ Coverage   95.78%   95.87%   +0.09%     
==========================================
  Files          65       66       +1     
  Lines        3745     3830      +85     
==========================================
+ Hits         3587     3672      +85     
  Misses        158      158              

Flag	Coverage Δ
	95.87% <100.00%> (+0.09%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[tatiana]

Hi @pankajkoti , great work!

Some minor feedback inline.

Something that I was wondering: I know we are currently not uploading the manifest file and other things from target, but we may want to do this in the future. Please, could you confirm that within the folder representing the dbt project, we have a target folder, and that compiled is a folder within it? This way, let's say we want to upload the manifest.json, we'd still be able to do this.

[pankajkoti]

Something that I was wondering: I know we are currently not uploading the manifest file and other things from target, but we may want to do this in the future. Please, could you confirm that within the folder representing the dbt project, we have a target folder, and that compiled is a folder within it? This way, let's say we want to upload the manifest.json, we'd still be able to do this.

Yes, previously, I was not creating a compiled subdirectory, but now I have added so. Also, as discussed made necessary changes to include the task_group if present in the identifier alongwith dag_id in the remote compiled sql file path.

[tatiana]

This looks great, @pankajkoti , thank you for addressing all the feedback. This is the first step of a very promising path, when Airflow worker nodes won't be blocked by running SQL transformations generated with dbt.

Something that we can look into as a follow-up is to introduce a teardown task to delete the dbt_compille artifacts. I logged a follow up task for us to look into this in the future: #1232 1232