-
Notifications
You must be signed in to change notification settings - Fork 159
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[DOCS]: Adds Welcome Section and "What Is Cosmos" Blurb to Home Page #1251
base: main
Are you sure you want to change the base?
Changes from 8 commits
8014500
2129294
61ab4dc
bc8d510
62ee76a
46c8981
61fa25d
a15f5db
cdfd348
555fbb4
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -28,19 +28,62 @@ | |||||
|
||||||
|fury| |ossrank| |downloads| |pre-commit| | ||||||
|
||||||
Run your dbt Core projects as `Apache Airflow® <https://airflow.apache.org/>`_ DAGs and Task Groups with a few lines of code. Benefits include: | ||||||
|
||||||
- Run dbt projects against Airflow connections instead of dbt profiles | ||||||
- Native support for installing and running dbt in a virtual environment to avoid dependency conflicts with Airflow | ||||||
- Run tests immediately after a model is done to catch issues early | ||||||
- Utilize Airflow's data-aware scheduling to run models immediately after upstream ingestion | ||||||
- Turn each dbt model into a task/task group complete with retries, alerting, etc. | ||||||
Welcome to Astronomer Cosmos! Whether you're an experienced data practitioner or just getting started, Cosmos makes it | ||||||
simple to manage and orchestrate your dbt workflows using `Apache Airflow® <https://airflow.apache.org/>`_, saving you | ||||||
time and effort. By automatically turning dbt workflows into Airflow DAGs, Cosmos allows you to focus on building | ||||||
high-quality data models without the hassle of managing complex integrations. | ||||||
|
||||||
To get started right away, please check out our `Quickstart Guides <https://astronomer.github.io/astronomer-cosmos/getting_started/index.html>`_. | ||||||
You can also explore more examples in `/dev/dags <https://github.com/astronomer/astronomer-cosmos/tree/main/dev/dags>`_ | ||||||
or in the `cosmos-demo repo <https://github.com/astronomer/cosmos-demo>`_. | ||||||
|
||||||
Example Usage | ||||||
___________________ | ||||||
To learn more about about Cosmos, please read on. | ||||||
|
||||||
You can render a Cosmos Airflow DAG using the ``DbtDag`` class. Here's an example with the `jaffle_shop project <https://github.com/dbt-labs/jaffle_shop>`_: | ||||||
|
||||||
What Is Astronomer Cosmos? | ||||||
___________________________ | ||||||
|
||||||
Astronomer Cosmos is an open-source library that bridges Apache Airflow and dbt, allowing you to easily transform your | ||||||
dbt projects into Airflow DAGs and manage everything seamlessly. With Cosmos, you can write your data transformations | ||||||
using dbt and then schedule and orchestrate them with Airflow, making the entire process smooth and straightforward. | ||||||
|
||||||
**Why Cosmos?** Integrating dbt and Airflow can be complex, but Cosmos simplifies it by seamlessly connecting these | ||||||
powerful tools—letting you focus on what matters most: delivering impactful data models and results without getting | ||||||
bogged down by technical challenges. | ||||||
|
||||||
|
||||||
Why Should You Use Cosmos? | ||||||
___________________________ | ||||||
|
||||||
Cosmos makes orchestrating dbt workflows: | ||||||
|
||||||
- **Effortless**: Transform your dbt projects into Airflow DAGs without writing extra code—Cosmos handles the heavy lifting. | ||||||
- **Reliable**: Rely on Airflow's robust scheduling and monitoring features to ensure your dbt workflows run smoothly and efficiently. | ||||||
- **Scalable**: Easily scale your workflows to match growing data demands, thanks to Airflow's distributed capabilities. | ||||||
|
||||||
Whether you're handling intricate data tasks or looking to streamline your processes, Cosmos helps you orchestrate dbt | ||||||
with Airflow effortlessly, saving you time and letting you focus on what truly matters—creating impactful insights. | ||||||
|
||||||
|
||||||
Example Usage: Jaffle Shop Project | ||||||
__________________________________ | ||||||
|
||||||
Let's explore a practical example to see how Cosmos can convert the dbt workflow into an Airflow DAG. | ||||||
|
||||||
The `jaffle_shop project <https://github.com/dbt-labs/jaffle_shop>`_ is a sample dbt project that simulates an e-commerce store's data. | ||||||
The project includes a series of dbt models that transform raw data into structured tables, such as sales, customers, and products. | ||||||
|
||||||
Below, you can see what the original dbt workflow looks like in a lineage graph. This graph helps illustrate the | ||||||
relationships between different models: | ||||||
|
||||||
.. image:: /_static/jaffle_shop_dbt_graph.png | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
wydt? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. When I tried that, it didn't render (I assume because it wasn't in version control yet). What's the difference between this, and using the relative path so that also works locally? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think they’re just alternatives to each other. I maybe wrong wrt rst syntax, however, I would consider There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. When I initially tried I then tried As written, the image shows up in the PR preview environment. See here - it's the image for the dbt lineage graph. I'm fine with switching to pulling the image from github main branch if that is what you want, but it means that this PR won't work until after the PR has been merged to main. |
||||||
|
||||||
Cosmos can take this dbt workflow and convert it into an Airflow DAG, allowing you to leverage Airflow's scheduling and | ||||||
orchestration capabilities. | ||||||
|
||||||
To convert this dbt workflow into an Airflow DAG, create a new DAG definition file, import ``DbtDag`` from the Cosmos library, | ||||||
and fill in a few parameters, such as the dbt project directory path and the profile name: | ||||||
|
||||||
.. | ||||||
The following renders in Sphinx but not Github: | ||||||
|
@@ -51,32 +94,42 @@ You can render a Cosmos Airflow DAG using the ``DbtDag`` class. Here's an exampl | |||||
:end-before: [END local_example] | ||||||
|
||||||
|
||||||
This will generate an Airflow DAG that looks like this: | ||||||
This code snippet will generate an Airflow DAG that looks like this: | ||||||
|
||||||
.. image:: https://raw.githubusercontent.com/astronomer/astronomer-cosmos/main/docs/_static/jaffle_shop_dag.png | ||||||
|
||||||
Getting Started | ||||||
_______________ | ||||||
``DbtDag`` is a custom DAG generator that converts dbt projects into Airflow DAGs and accepts Cosmos-specific args like | ||||||
``fail_fast`` to immediately fail a dag if dbt fails to process a resource, or ``cancel_query_on_kill`` to cancel any running | ||||||
queries if the task is externally killed or manually set to failed in Airflow. ``DbtDag`` also accepts standard DAG arguments such | ||||||
as ``max_active_tasks``, ``max_active_runs`` and ``default_args``. | ||||||
|
||||||
Check out the Quickstart guide on our `docs <https://astronomer.github.io/astronomer-cosmos/getting_started/index.html>`_. See more examples at `/dev/dags <https://github.com/astronomer/astronomer-cosmos/tree/main/dev/dags>`_ and at the `cosmos-demo repo <https://github.com/astronomer/cosmos-demo>`_. | ||||||
With Cosmos, transitioning from a dbt workflow to a proper Airflow DAG is seamless, giving you the best of both tools | ||||||
for managing and scaling your data workflows. | ||||||
|
||||||
|
||||||
Changelog | ||||||
_________ | ||||||
|
||||||
We follow `Semantic Versioning <https://semver.org/>`_ for releases. | ||||||
Check `CHANGELOG.rst <https://github.com/astronomer/astronomer-cosmos/blob/main/CHANGELOG.rst>`_ | ||||||
Refer to `CHANGELOG.rst <https://github.com/astronomer/astronomer-cosmos/blob/main/CHANGELOG.rst>`_ | ||||||
for the latest changes. | ||||||
|
||||||
Contributing Guide | ||||||
|
||||||
Join the Community | ||||||
__________________ | ||||||
|
||||||
All contributions, bug reports, bug fixes, documentation improvements, enhancements are welcome. | ||||||
Have questions, need help, or interested in contributing? We welcome all contributions and feedback! | ||||||
|
||||||
- Join the community on Slack! You can find us in the Airflow Slack workspace `#airflow-dbt <https://apache-airflow.slack.com/archives/C059CC42E9W>`_ | ||||||
channel. If you don't have an account, click `here <https://apache-airflow-slack.herokuapp.com/>`_ to sign up. | ||||||
cmarteepants marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
- Report bugs, request features, or ask questions by creating an issue in the `GitHub repository <https://github.com/astronomer/astronomer-cosmos/issues/new/choose>`_. | ||||||
|
||||||
- Want to contribute new features, bug fixes or documentation enhancements? Please refer to our `Contributing Guide <https://astronomer.github.io/astronomer-cosmos/contributing>`_. | ||||||
|
||||||
A detailed overview on how to contribute can be found in the `Contributing Guide <https://astronomer.github.io/astronomer-cosmos/contributing>`_. | ||||||
Find out more about `our contributors <https://astronomer.github.io/astronomer-cosmos/contributors>`_. | ||||||
- Check out this `link <https://astronomer.github.io/astronomer-cosmos/contributors>`_. to learn more about our current contributors | ||||||
cmarteepants marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
As contributors and maintainers to this project, you are expected to abide by the | ||||||
Note that contributors and maintainers are expected to abide by the | ||||||
`Contributor Code of Conduct <https://github.com/astronomer/astronomer-cosmos/blob/main/CODE_OF_CONDUCT.md>`_. | ||||||
|
||||||
|
||||||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shall we use "Dbt-core" instead of "dbt"?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@tatiana / @pankajkoti any thoughts ^^?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
not too sure. Maybe we could add a line somewhere that in general when we mean
dbt
, we're referring todbt-core
workflows throughout the references. no strong opinion.