Release 1.0.5 #453
Merged
Release 1.0.5 #453
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
## Description <!-- Add a brief but complete description of the change. --> This PR updates the docs site with a better structure. It also adds getting started guides for each of the popular ways of running Airflow. ## Related Issue(s) Closes: #219 Closes: #316 Closes: #307 ## Breaking Change? <!-- If this introduces a breaking change, specify that here. --> ## Checklist - [ ] I have made corresponding changes to the documentation (if required) - [ ] I have added tests that prove my fix is effective or that my feature works --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Harel Shein <[email protected]>
## Description <!-- Add a brief but complete description of the change. --> ## Related Issue(s) <!-- If this PR closes an issue, you can use a keyword to auto-close. --> <!-- i.e. "closes #0000" --> ## Breaking Change? <!-- If this introduces a breaking change, specify that here. --> ## Checklist - [ ] I have made corresponding changes to the documentation (if required) - [ ] I have added tests that prove my fix is effective or that my feature works
## Description In the process of setting up the dev environment, an empty .env file is requested to build the image. Since there is no visible impact on having it not having it, for consistency it is better to remove this not used dependency from the docker compose yml file. ## Related Issue(s) somehow related to #342 ## Breaking Change? No major changes are going to happen ## Checklist - [ ] I have made corresponding changes to the documentation (if required) - [ ] I have added tests that prove my fix is effective or that my feature works --------- Co-authored-by: giovanni.corsetti <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Harel Shein <[email protected]>
## Description When using a virtualenv to run our dbt tasks, we are working under the hypothesis that dbt is not installed in the machine. However as of now, if we choose virtualenv as the execution mode, dbt tries to parse the graph of the dbt tasks with **load_via_dbt_ls()**, while it should point out to **load_via_custom_parser()** due to the fact aforementioned. we can accomplish it by removing the virutalenv executor from the _elif_ condition for **load_via_dbt_ls()**, so it will be picked up automatically by the _else_ condition. ## Related Issue(s) closes #401 ## Breaking Change? No. it worked fine in the dev sandbox, although I did not have the opportunity to test it with any image without dbt pre-installed. Obs: I had to change also the tests for this change ## Checklist - [X] I have made corresponding changes to the documentation (if required) - [ ] I have added tests that prove my fix is effective or that my feature works --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Julian LaNeve <[email protected]>
## Description When attempting to override the `group_id` parameter of a DbtTaskGroup, I get the following error: ```text TypeError: __init__() got multiple values for argument 'group_id' ``` ## Related Issue(s) closes #398 ## Breaking Change? There are no breaking changes with this PR ## Checklist - [x] I have made corresponding changes to the documentation (if required) - [x] I have added tests that prove my fix is effective or that my feature works Co-authored-by: Julian LaNeve <[email protected]>
## Description <!-- Add a brief but complete description of the change. --> Before, profile mappings would override `__getattr__` to get the dbt profile value, primarily for convenience. However, this was causing unintended side effects - infinite recursive loops. This PR fixes the issue by removing the `__getattr__` override in favor of being more explicit. ## Related Issue(s) <!-- If this PR closes an issue, you can use a keyword to auto-close. --> <!-- i.e. "closes #0000" --> ## Breaking Change? <!-- If this introduces a breaking change, specify that here. --> No, this is all behind-the-scenes. ## Checklist - [ ] I have made corresponding changes to the documentation (if required) - [ ] I have added tests that prove my fix is effective or that my feature works
## Description When `load_method` was set to `LoadMode.CUSTOM`, any `select/exclude` parameters were ineffective. This is because the entire node list was being passed to the `build_airflow_graph` method instead of a filtered list of nodes. ## Related Issue(s) closes #409 ## Breaking Change? No breaking changes with this change ## Checklist - [x] I have made corresponding changes to the documentation (if required) - [ ] I have added tests that prove my fix is effective or that my feature works
## Description Rename docs to match code ## Related Issue(s) closes #412 ## Breaking Change? no. ## Checklist - [x] I have made corresponding changes to the documentation (if required) - [x] I have added tests that prove my fix is effective or that my feature works
<!--pre-commit.ci start--> updates: - [github.com/asottile/pyupgrade: v3.9.0 → v3.10.1](asottile/pyupgrade@v3.9.0...v3.10.1) - [github.com/astral-sh/ruff-pre-commit: v0.0.280 → v0.0.281](astral-sh/ruff-pre-commit@v0.0.280...v0.0.281) <!--pre-commit.ci end--> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
## Description
In the documentation of astronomer, user-supplied profiles are defined
as follows
```
from cosmos.config import ProfileConfig
profile_config = ProfileConfig(
profile_name="my_snowflake_profile",
target_name="dev",
profiles_yml_filepath="/path/to/profiles.yml",
)
dag = DbtDag(profile_config=profile_config, ...)
```
however in the code the _profiles_yml_filepath_ crashes if a string is
supplied to it, since it assumes a Path type instead of a str type. In
the code logic, **self.profiles_yml_filepath.exists()** crashes if the
profiles_yml_filepath is a str. this PR just makes a few adjustment to
allow users to supply a custom filepath in a string format without
having to wrap it over a Path()
## Related Issue(s)
---
## Breaking Change?
No
## Checklist
- [X] I have made corresponding changes to the documentation (if
required)
- [ ] I have added tests that prove my fix is effective or that my
feature works
---------
Co-authored-by: giovanni.corsetti <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Harel Shein <[email protected]>
## Description <!-- Add a brief but complete description of the change. --> The Kubernetes Airflow provider and dbt Core released new versions in the last 24 hours, both of which broke Cosmos. This puts an upper version limit on the packages while we work to address those breaking changes. ## Related Issue(s) <!-- If this PR closes an issue, you can use a keyword to auto-close. --> <!-- i.e. "closes #0000" --> ## Breaking Change? <!-- If this introduces a breaking change, specify that here. --> No ## Checklist - [ ] I have made corresponding changes to the documentation (if required) - [ ] I have added tests that prove my fix is effective or that my feature works
## Description Simply adds "source" as a DbtResourceType. ## Checklist - [ ] I have made corresponding changes to the documentation (if required) - [ ] I have added tests that prove my fix is effective or that my feature works
## Description <!-- Add a brief but complete description of the change. --> ## Related Issue(s) <!-- If this PR closes an issue, you can use a keyword to auto-close. --> <!-- i.e. "closes #0000" --> ## Breaking Change? <!-- If this introduces a breaking change, specify that here. --> ## Checklist - [ ] I have made corresponding changes to the documentation (if required) - [ ] I have added tests that prove my fix is effective or that my feature works
## Description Throws an error if select/exclude filters are not valid. Validation: Must start with "path:", "tag:", or "config.materialized", "config.schema", or "config.tags" ## Checklist - [ ] I have made corresponding changes to the documentation (if required) - [ ] I have added tests that prove my fix is effective or that my feature works
## Description Support OAuth authentication for Big Query ## Related Issue(s) closes #420 ## Breaking Change? None ## Checklist - [ ] I have made corresponding changes to the documentation (if required) - [X] I have added tests that prove my fix is effective or that my feature works --------- Co-authored-by: Monideep De <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
## Description Provided different solutions for new MyPy errors that were the result of adding "apache-airflow" as a dependency within the pre-commit config. ## Checklist - [ ] I have made corresponding changes to the documentation (if required) - [ ] I have added tests that prove my fix is effective or that my feature works --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
<!--pre-commit.ci start--> updates: - [github.com/Lucas-C/pre-commit-hooks: v1.5.1 → v1.5.3](Lucas-C/pre-commit-hooks@v1.5.1...v1.5.3) - [github.com/astral-sh/ruff-pre-commit: v0.0.281 → v0.0.282](astral-sh/ruff-pre-commit@v0.0.281...v0.0.282) <!--pre-commit.ci end--> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Previously, we were using Airflow standard logging format for Cosmos messages, which looks like:
```
[2023-08-09T14:20:55.532+0100] {subprocess.py:94} INFO - 13:20:55 Completed successfully
```
This change introduces a (yellow) "astronomer-cosmos" string into the
project's log messages:
```
[2023-08-09T14:20:55.532+0100] {subprocess.py:94} INFO - (astronomer-cosmos) - 13:20:55 Completed successfully
```
It also adds more logs to DBT project parsing to help troubleshoot
issues while rendering the DAG.
Example of how the output looks like:
Closes: #447
…rver (#449) GCP connections created using a key file JSON via the webserver UI were not previously working, although the connections created via the command line worked as expected. The reason for this bug is that when the UI is used, the `keyfile_dict` field is stored as a string. Closes: #443 Closes: #436
Fixing an issue where the check in `selector.py`, specifically `select_nodes_ids_by_intersection`, didn't validate tags in the node's config. The function will now check if the tag specified in config is contained in the node's config and check all other configs to be filtered for are a subset of the node's config, as previously. Closes: #440
Enhancements * Improve logs to include astornomer-cosmos identifier by @tatiana in #450 * Support OAuth authentication for Big Query by @MonideepDe in #431 Bug fixes * Fix selector for config tags by @javihernovoa in #441 * Fix BigQuery keyfile_dict mapping for connection created from webserver UI by @jbandoro in #449 Others * [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #446 * Resolve MyPy errors when adding Airflow pre-commit dependency by @abhi12mohan in #434
👷 Deploy Preview for amazing-pothos-a3bca0 processing.
|
Codecov ReportPatch coverage:
Additional details and impacted files@@ Coverage Diff @@
## main #453 +/- ##
=======================================
Coverage 91.43% 91.43%
=======================================
Files 50 50
Lines 1752 1752
=======================================
Hits 1602 1602
Misses 150 150
☔ View full report in Codecov by Sentry. |
Fix dead links in the live docs: * Source: https://astronomer.github.io/astronomer-cosmos/getting_started/execution-modes.html * Previous target: https://astronomer.github.io/astronomer-cosmos/getting_started/execution-mode-local-conflicts.html Restore the compatibility matrix between Airflow and dbt, originally introduced in: #347 This change was validated using the following steps: * Locally, using ` hatch run docs:build` and confirming using Chrome that the generated HTML `file:///<path>/astronomer-cosmos/docs/_build/dbt/execution-modes.html` resolved correctly the path to the conflicts page * Remotely, confirming that https://deploy-preview-452--amazing-pothos-a3bca0.netlify.app/getting_started/execution-modes/ conflict link is associated to https://deploy-preview-452--amazing-pothos-a3bca0.netlify.app/getting_started/execution-modes-local-conflicts/#execution-modes-local-conflicts <img width="1280" alt="Screenshot 2023-08-09 at 23 12 53" src="https://github.com/astronomer/astronomer-cosmos/assets/272048/6b6da81e-4d88-48b4-a712-9bed95ae93a4">
Before 1.0 this was valid:
```
from cosmos.task_group import DbtTaskGroup
```
Now we expect users to do:
```
from cosmos import DbtTaskGroup
```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Enhancements
Bug fixes
keyfile_dictmapping for connection created from webserver #449Others