Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support Python models in Cosmos #182

Closed
aiyaswamy-tgam opened this issue Mar 9, 2023 · 3 comments · Fixed by #375
Closed

Support Python models in Cosmos #182

aiyaswamy-tgam opened this issue Mar 9, 2023 · 3 comments · Fixed by #375
Assignees
@tatiana
Labels
team/astro-devex
Milestone
1.0.0

Comments

@aiyaswamy-tgam
Copy link

aiyaswamy-tgam commented Mar 9, 2023 •
edited by harels
Loading

Background:
DBT support Python models as well, we would like to make sure Cosmos supports this functionality in DBT.
DbtTaskGroup supports only the SQL files in the render class, Does not support python dbt models

Acceptance Criteria:
We can run Python models in Cosmos (in all execution modes?)
@chrishronek
Copy link
Contributor

@aiyaswamy-tgam by Python dbt models, you mean something like this, correct?

@chrishronek chrishronek self-assigned this Mar 17, 2023
@chrishronek chrishronek added provider/dbt priority:high High priority issues are blocking or critical issues without a workaround and large impact and removed provider/dbt labels Mar 17, 2023
@aiyaswamy-tgam
Copy link
Author

@aiyaswamy-tgam by Python dbt models, you mean something like this, correct?

Yes, that's right!

@rightx2
Copy link

rightx2 commented Mar 28, 2023

I need this function as well.

@chrishronek chrishronek added this to the 1.0.0 milestone Apr 18, 2023
@jlaneve jlaneve changed the title Task group DOES NOT support python dbt models Support Python models in Cosmos Jun 6, 2023
@JoePollo JoePollo mentioned this issue Jun 12, 2023
Closed
@jlaneve jlaneve removed the priority:high High priority issues are blocking or critical issues without a workaround and large impact label Jun 20, 2023
tatiana added a commit that referenced this issue Jul 19, 2023
@tatiana
Add support to Python models when using LoadMode.DBT_LS
78a7d31 
Relates to: #182
@tatiana tatiana self-assigned this Jul 19, 2023
@tatiana tatiana mentioned this issue Jul 19, 2023
Merged
tatiana added a commit that referenced this issue Jul 23, 2023
@tatiana
Add support to Python models when using LoadMode.DBT_LS
5e697f0 
Relates to: #182
tatiana added a commit that referenced this issue Jul 23, 2023
@tatiana
Add support to run dbt Python models (#375)
1058544 
Add support to run dbt Python models, as described in the official
documentation: https://docs.getdbt.com/docs/build/python-models

Add an example of Cosmos running Python models on Databricks (Postgres
is not supported as of dbt 1.6).

The dbt example can be run by itself from the directory
`dev/dags/dbt/jaffle_shop_python` by exporting the following environment
variables:
* DATABRICKS_HOST: Databricks host, similar to:
dbc-some-id.cloud.databricks.com
* DATABRICKS_WAREHOUSE_ID: Databricks SQL Warehouse ID from connection
HTTP path (example: `ca312a2206dfb361`)
* DATABRICKS_TOKEN: User Databricks access token
* DATABRICKS_CLUSTER_ID: Databricks cluster ID (example:
0201-094352-wab833sb)

And running:
```
dbt build
```

To validate the feature from a Cosmos perspective, set up the
`databrics_default` connection. One way of accomplishing this is by
using environment variables:
```
export AIRFLOW_CONN_DATABRICKS_DEFAULT=databricks://@dbc-<account-id>.cloud.databricks.com?token=<access-token>&http_path=/sql/1.0/warehouses/<warehouse-id>
```

From a previously set-up Airflow environment, run the
`example_cosmos_python_models` DAG. An example of how to execute it from
the command line:
```
airflow dags test example_cosmos_python_models `date -Iseconds`
```

This feature was validated with `load_mode=LoadMode.DBT_LS` and
`LoadMode.CUSTOM`. Example of the rendered DAG:
<img width="1240" alt="Screenshot 2023-07-19 at 23 46 14"
src="https://github.com/astronomer/astronomer-cosmos/assets/272048/6394fe31-30ad-4635-b9dd-f7172929aaa5">

Review can be simplified by checking the commits individually,
especially between 796d6e8 and
78a7d31.

The downside of this change is that the integration tests are now
slower to run and depend on the following environment variables being
set up in the CI:
* `AIRFLOW_CONN_DATABRICKS_DEFAULT`
* `DATABRICKS_CLUSTER_ID`

Closes: #182
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tatiana tatiana

Labels
team/astro-devex
Projects
None yet
Milestone
1.0.0
Development

Successfully merging a pull request may close this issue.

Add support to run dbt Python models astronomer/astronomer-cosmos
6 participants
@tatiana @harels @jlaneve @rightx2 @chrishronek @aiyaswamy-tgam