Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Release Cosmos 1.6.0 #1103

Closed
18 tasks done
phanikumv opened this issue Jul 18, 2024 · 6 comments
Closed
18 tasks done

Release Cosmos 1.6.0 #1103

phanikumv opened this issue Jul 18, 2024 · 6 comments
Assignees
@tatiana @pankajkoti
Labels
epic Q3
Milestone
Cosmos 1.6.0

Comments

@phanikumv
Copy link
Collaborator

phanikumv commented Jul 18, 2024 •
edited by pankajkoti
Loading

@phanikumv phanikumv added enhancement New feature or request triage-needed Items need to be reviewed / assigned to milestone epic labels Jul 18, 2024
@phanikumv phanikumv added this to the Cosmos 1.6.0 milestone Jul 18, 2024
@phanikumv phanikumv removed enhancement New feature or request triage-needed Items need to be reviewed / assigned to milestone labels Jul 18, 2024
@phanikumv phanikumv added the Q3 label Jul 25, 2024
@tatiana
Copy link
Collaborator

tatiana commented Aug 1, 2024 •
edited
Loading

I updated the description of this release to match the description of the milestone we had created.

This is the list of tasks:

@pankajkoti pankajkoti self-assigned this Aug 12, 2024
@phanikumv
Copy link
Collaborator Author

Can we check the scope again for 1.6.0 please? Are we on track to complete the above tasks

cc @tatiana

@tatiana
Copy link
Collaborator

tatiana commented Aug 14, 2024

Description authored by @tatiana @pankajastro @pankajkoti

Many users from OSS and Astronomer raised issues related to disk utilization. In the past (until Cosmos 1.3.x), these were exclusive to LoadMode.VIRTUALENV (#958). Still, more recently (since Cosmos 1.4.x), with the introduction of local cache, we intentionally started caching in the local temporary directory, leading to issues (#1042). From an Astro cloud perspective, the introduction of Ephemeral storage has caused this issue to become even more evident. For this reason, actions that aim to reduce the local cache are a priority in 1.6.

We aim to address these by closing the following tickets, which we consider critical:

In the meantime, as of Cosmos 1.5, we introduced DBT_LS cache into Airflow variables. We have customers using this feature in production, but some of them would like to store this cache in a remote location. We agreed with them to address this by 1.6:

One of the most popular community and customer requests is:

Finally, another extremely popular request has been for Cosmos to have a default way to handle dbt Source nodes. There are several open tickets related to this, and also a PR from a contributor who happens to be an Astronomer customer:

Now is the right time to talk about this.
Our integration tests do not run DAGs using the ExecutionMode.KUBERNETES. This means we rely on human testing, which sometimes may be forgotten and can incur into issues. The following ticket aims to address this:
#535

Finally, there are some other tasks that we're aiming to wrap up:

@pankajkoti
Copy link
Contributor

We created an alpha7 last Friday with all the PRs merged as reflected in the updated issue description.

We're testing the alpha7 at the moment.

We likely would have to release another alpha8 including the PR #1162 and test that together with other changes before releasing 1.6.0

@pankajkoti
Copy link
Contributor

We've released Cosmos 1.6.0 via #1080.

@phanikumv should we close this issue (labelled as epic) or keep it open?

@phanikumv
Copy link
Collaborator Author

@pankajkoti We can close this issue, great work on delivering the release 🎉

@pankajkoti pankajkoti mentioned this issue Aug 29, 2024
Merged
2 tasks
pankajkoti added a commit that referenced this issue Aug 29, 2024
@pankajkoti
Resolve CI failures for Apache Airflow 2.7 jobs (#1182)
9b889a4 
## Description

This is an effort to fix the currently broken CI jobs. The CI pipeline
was running smoothly until the Cosmos 1.6.0 release (#1103). However,
after the recent release of Apache Airflow providers, we’ve been
encountering deep resolution errors specifically with Apache Airflow 2.7
jobs.

To address this:

1. For Airflow 2.7 tests, we’re pre-installing the providers using
Airflow 2.7 constraints.
2. We’re also upgrading certain jobs, like type-checks and dbt 1.5.4
tests, from Airflow 2.7 to Airflow 2.8, as it seems beneficial to move
from 2.7 to 2.8.

Despite pre-installing the providers, we’re noticing that on Python 3.8
and 3.9 with Apache Airflow 2.7, Cosmos still attempts to sync
dependencies and continues to encounter deep resolution errors.
Therefore, we’re specifically excluding those combinations from running
in Unit and Integration test jobs. However, we have Python 3.10 and 3.11
jobs enabled and running fine for Airflow 2.7 in our CI.

## Related Issue(s)
related: #1180
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tatiana tatiana

@pankajkoti pankajkoti

Labels
epic Q3
Projects
None yet
Milestone
Cosmos 1.6.0
Development

No branches or pull requests
3 participants
@tatiana @pankajkoti @phanikumv