Consistently handle build and imports in cosmos/init.py #1215

tatiana · 2024-09-23T07:01:53Z

While reviewing #1153, I noticed we were inconsistent with Build operators, especially how we import and expose them in Cosmos __init__.py This PR addresses this.

These were the changes introduced:

Add DbtBuildAzureContainerInstanceOperator
Expose the build operators for all execution modes in Cosmos __init__ (missing for most, except ExecutionMode.LOCAL)
Be consistent with import error messages in __init__ (we were missing AWS EKS)

…n Cosmos __init__, including missing EKS and a few Build ones

netlify · 2024-09-23T07:02:11Z

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name	Link
🔨 Latest commit	`3ca4e0a`
🔍 Latest deploy log	https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/66f112632916f1000811d765

pankajastro

LGTM

codecov · 2024-09-23T08:07:40Z

Codecov Report

Attention: Patch coverage is 43.75000% with 9 lines in your changes missing coverage. Please review.

Project coverage is 95.72%. Comparing base (11de5ba) to head (3ca4e0a).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
cosmos/__init__.py	25.00%	9 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1215      +/-   ##
==========================================
- Coverage   95.95%   95.72%   -0.23%     
==========================================
  Files          64       64              
  Lines        3656     3672      +16     
==========================================
+ Hits         3508     3515       +7     
- Misses        148      157       +9

Flag	Coverage Δ
	`95.72% <43.75%> (-0.23%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

pankajastro · 2024-09-23T08:13:15Z

Not in this PR but @tatiana do you think it makes sense to add some script in pre-commit to check the class from the constant and operator module included in __init__ file?

pankajastro · 2024-09-23T08:15:02Z

There was some similar fix #1184 in the past also but looks like we missed some

tatiana · 2024-09-23T08:29:20Z

Thanks for the light-of-speed review, @pankajastro !

It's not in this PR, but @tatiana, do you think it makes sense to add a script in the pre-commit to check the class from the constant and operator module included in the init file?
Yes, I believe that it makes sense to automate the check for imports, I logged a ticket so we don't forget this: #1216

In the mid-long term, I'd love to stop this practice of having everything exposed in cosmos/__init__.py. People could just import them from other locations. I logged a ticket for this: #1213

@pankajkoti

New Features * Introduction of experimental support to run dbt BQ models using Airflow deferrable operators by @pankajkoti @pankajastro @tatiana in #1224 #1230. This is a first step in this journey and we would really appreciate feedback from the community. For more information, check the documentation: https://astronomer.github.io/astronomer-cosmos/getting_started/execution-modes.html#airflow-async-experimental This work has been inspired by the talk "Airflow at Monzo: Evolving our data platform as the bank scales" by @jonathanrainer @ed-sparkes given at Airflow Summit 2023: https://airflowsummit.org/sessions/2023/airflow-at-monzo-evolving-our-data-platform-as-the-bank-scales/. * Support using ``DatasetAlias`` and fix orphaning unreferenced dataset by @tatiana in #1217 #1240 Documentation: https://astronomer.github.io/astronomer-cosmos/configuration/scheduling.html#data-aware-scheduling * Add GCP_CLOUD_RUN_JOB execution mode by @ags-de #1153 Learn more about it: https://astronomer.github.io/astronomer-cosmos/getting_started/gcp-cloud-run-job.html Enhancements * Create single virtualenv when ``DbtVirtualenvBaseOperator`` has ``virtualenv_dir=None`` and ``is_virtualenv_dir_temporary=True`` by @kesompochy in #1200 * Consistently handle build and imports in ``cosmos/__init__.py`` by @tatiana in #1215 * Add enum constants to init for direct import by @fabiomx in #1184 Bug fixes * URL encode dataset names to support multibyte characters by @t0momi219 in #1198 * Fix invalid argument (``full_refresh``) passed to DbtTestAwsEksOperator (and others) by @johnhoran in #1175 * Fix ``printer_width`` arg type in ``DbtProfileConfigVars`` by @jessicaschueler in #1191 * Fix task owner fallback by @jmaicher in #1195 Docs * Add scarf to readme and docs for website analytics by @cmarteepants in #1221 * Add ``virtualenv_dir`` param to ``ExecutionConfig`` docs by @pankajkoti in #1173 * Give credits to @LennartKloppenburg in CHANGELOG.rst by @tatiana #1174 * Refactor docs for async mode execution by @pankajkoti in #1241 Others * Remove PR branch added for testing a change in CI in #1224 by @pankajkoti in #1233 * Fix CI wrt broken coverage upload artifact @pankajkoti in #1210 * Fix CI issues - Upgrade actions/upload-artifact & actions/download-artifact to v4 and set min version for packaging by @pankajkoti in #1208 * Resolve CI failures for Apache Airflow 2.7 jobs by @pankajkoti in #1182 * CI: Update GCP manifest file path based on new secret update by @pankajkoti in #1237 * Pre-commit hook updates in #1176 #1186, #1186, #1201, #1219, #1231

@pankajkoti

New Features * Introduction of experimental support to run dbt BQ models using Airflow deferrable operators by @pankajkoti @pankajastro @tatiana in #1224 #1230. This is a first step in this journey and we would really appreciate feedback from the community. For more information, check the documentation: https://astronomer.github.io/astronomer-cosmos/getting_started/execution-modes.html#airflow-async-experimental This work has been inspired by the talk "Airflow at Monzo: Evolving our data platform as the bank scales" by @jonathanrainer @ed-sparkes given at Airflow Summit 2023: https://airflowsummit.org/sessions/2023/airflow-at-monzo-evolving-our-data-platform-as-the-bank-scales/. * Support using ``DatasetAlias`` and fix orphaning unreferenced dataset by @tatiana in #1217 #1240 Documentation: https://astronomer.github.io/astronomer-cosmos/configuration/scheduling.html#data-aware-scheduling * Add GCP_CLOUD_RUN_JOB execution mode by @ags-de #1153 Learn more about it: https://astronomer.github.io/astronomer-cosmos/getting_started/gcp-cloud-run-job.html Enhancements * Create single virtualenv when ``DbtVirtualenvBaseOperator`` has ``virtualenv_dir=None`` and ``is_virtualenv_dir_temporary=True`` by @kesompochy in #1200 * Consistently handle build and imports in ``cosmos/__init__.py`` by @tatiana in #1215 * Add enum constants to init for direct import by @fabiomx in #1184 Bug fixes * URL encode dataset names to support multibyte characters by @t0momi219 in #1198 * Fix invalid argument (``full_refresh``) passed to DbtTestAwsEksOperator (and others) by @johnhoran in #1175 * Fix ``printer_width`` arg type in ``DbtProfileConfigVars`` by @jessicaschueler in #1191 * Fix task owner fallback by @jmaicher in #1195 Docs * Add scarf to readme and docs for website analytics by @cmarteepants in #1221 * Add ``virtualenv_dir`` param to ``ExecutionConfig`` docs by @pankajkoti in #1173 * Give credits to @LennartKloppenburg in CHANGELOG.rst by @tatiana #1174 * Refactor docs for async mode execution by @pankajkoti in #1241 Others * Remove PR branch added for testing a change in CI in #1224 by @pankajkoti in #1233 * Fix CI wrt broken coverage upload artifact @pankajkoti in #1210 * Fix CI issues - Upgrade actions/upload-artifact & actions/download-artifact to v4 and set min version for packaging by @pankajkoti in #1208 * Resolve CI failures for Apache Airflow 2.7 jobs by @pankajkoti in #1182 * CI: Update GCP manifest file path based on new secret update by @pankajkoti in #1237 * Pre-commit hook updates in #1176 #1186, #1186, #1201, #1219, #1231

@tatiana

**New Features** * Support using ``DatasetAlias`` and fix orphaning unreferenced dataset by @tatiana in #1217 #1240 Documentation: https://astronomer.github.io/astronomer-cosmos/configuration/scheduling.html#data-aware-scheduling * Add GCP_CLOUD_RUN_JOB execution mode by @ags-de #1153 Learn more about it: https://astronomer.github.io/astronomer-cosmos/getting_started/gcp-cloud-run-job.html * Introduction of experimental support to run dbt BQ models using Airflow deferrable operators by @pankajkoti @pankajastro @tatiana in #1224 #1230. This is the first step in the journey of running dbt resources with native Airflow, and we would appreciate feedback from the community. For more information, check the documentation: https://astronomer.github.io/astronomer-cosmos/getting_started/execution-modes.html#airflow-async-experimental This work has been inspired by the talk "Airflow at Monzo: Evolving our data platform as the bank scales" by @jonathanrainer @ed-sparkes given at Airflow Summit 2023: https://airflowsummit.org/sessions/2023/airflow-at-monzo-evolving-our-data-platform-as-the-bank-scales/. **Enhancements** * Create single virtualenv when ``DbtVirtualenvBaseOperator`` has ``virtualenv_dir=None`` and ``is_virtualenv_dir_temporary=True`` by @kesompochy in #1200 * Consistently handle build and imports in ``cosmos/__init__.py`` by @tatiana in #1215 * Add enum constants to init for direct import by @fabiomx in #1184 **Bug fixes** * URL encode dataset names to support multibyte characters by @t0momi219 in #1198 * Fix invalid argument (``full_refresh``) passed to DbtTestAwsEksOperator (and others) by @johnhoran in #1175 * Fix ``printer_width`` arg type in ``DbtProfileConfigVars`` by @jessicaschueler in #1191 * Fix task owner fallback by @jmaicher in #1195 **Docs** * Add scarf to readme and docs for website analytics by @cmarteepants in #1221 * Add ``virtualenv_dir`` param to ``ExecutionConfig`` docs by @pankajkoti in #1173 * Give credits to @LennartKloppenburg in CHANGELOG.rst by @tatiana #1174 * Refactor docs for async mode execution by @pankajkoti in #1241 Others * Remove PR branch added for testing a change in CI in #1224 by @pankajkoti in #1233 * Fix CI wrt broken coverage upload artifact @pankajkoti in #1210 * Fix CI issues - Upgrade actions/upload-artifact & actions/download-artifact to v4 and set min version for packaging by @pankajkoti in #1208 * Resolve CI failures for Apache Airflow 2.7 jobs by @pankajkoti in #1182 * CI: Update GCP manifest file path based on new secret update by @pankajkoti in #1237 * Pre-commit hook updates in #1176 #1186, #1186, #1201, #1219, #1231 --------- Co-authored-by: Pankaj Koti <[email protected]>

Add DbtBuildAzureContainerInstanceOperator and expose all operators i…

3ca4e0a

…n Cosmos __init__, including missing EKS and a few Build ones

tatiana temporarily deployed to internal September 23, 2024 07:01 — with GitHub Actions Inactive

tatiana marked this pull request as ready for review September 23, 2024 07:49

dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. area:execution Related to the execution environment/mode, like Docker, Kubernetes, Local, VirtualEnv, etc dbt:build Primarily related to dbt build command or functionality labels Sep 23, 2024

tatiana mentioned this pull request Sep 23, 2024

Add GCP_CLOUD_RUN_JOB execution mode #1153

Merged

2 tasks

pankajastro approved these changes Sep 23, 2024

View reviewed changes

dosubot bot added the lgtm This PR has been approved by a maintainer label Sep 23, 2024

tatiana mentioned this pull request Sep 23, 2024

Add script in pre-commit to check the operator and constant classes are in __init__ #1216

Open

tatiana merged commit 0fd0bdd into main Sep 23, 2024
66 of 68 checks passed

tatiana deleted the add-build-operators-to-import branch September 23, 2024 08:29

tatiana added this to the Cosmos 1.7.0 milestone Sep 23, 2024

pankajkoti approved these changes Sep 23, 2024

View reviewed changes

tatiana mentioned this pull request Oct 2, 2024

Release 1.7 #1234

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Consistently handle build and imports in cosmos/init.py #1215

Consistently handle build and imports in cosmos/init.py #1215

tatiana commented Sep 23, 2024

netlify bot commented Sep 23, 2024 •

edited

Loading

pankajastro left a comment

codecov bot commented Sep 23, 2024 •

edited

Loading

pankajastro commented Sep 23, 2024

pankajastro commented Sep 23, 2024

tatiana commented Sep 23, 2024

Consistently handle build and imports in cosmos/__init__.py #1215

Consistently handle build and imports in cosmos/__init__.py #1215

Conversation

tatiana commented Sep 23, 2024

netlify bot commented Sep 23, 2024 • edited Loading

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

pankajastro left a comment

Choose a reason for hiding this comment

codecov bot commented Sep 23, 2024 • edited Loading

Codecov Report

pankajastro commented Sep 23, 2024

pankajastro commented Sep 23, 2024

tatiana commented Sep 23, 2024

Consistently handle build and imports in cosmos/init.py #1215

Consistently handle build and imports in cosmos/init.py #1215

netlify bot commented Sep 23, 2024 •

edited

Loading

codecov bot commented Sep 23, 2024 •

edited

Loading