Update Airflow Astro deployment docs #3792

DimedS · 2024-04-08T15:49:56Z

Description

This PR addresses issues 1, 2, 3, 5, and 6 from 605:

The architecture of the Astro deployment manual has been restructured into two stages: preparation of the Kedro project and Astro deployment. Redundancies have been eliminated, and the manual is now updated and fully operational.
The kedro-airflow-k8s plugin has been relocated to the end of the document because it is compatible only with Kedro versions earlier than 0.17.
The astro-airflow-iris starter has been replaced with the standard spaceflights-pandas example pipeline.
Customisation of logging has been introduced.
Instructions for transferring results back from the container have been added.

Developer Certificate of Origin

We need all contributions to comply with the Developer Certificate of Origin (DCO). All commits must be signed off by including a Signed-off-by line in the commit message. See our wiki for guidance.

If your PR is blocked due to unsigned commits, then you must follow the instructions under "Rebase the branch" on the GitHub Checks page for your PR. This will retroactively add the sign-off to all unsigned commits and allow the DCO check to pass.

Checklist

Read the contributing guidelines
Signed off each commit with a Developer Certificate of Origin (DCO)
Opened this PR as a 'Draft Pull Request' if it is work-in-progress
Updated the documentation to reflect the code changes
Added a description of this change in the RELEASE.md file
Added tests to cover my changes
Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team

Signed-off-by: Dmitry Sorokin <[email protected]>

…manual

Signed-off-by: Dmitry Sorokin <[email protected]>

merelcht

This is a fantastic start @DimedS ⭐ I have left some comments inline, but in general I think it would be useful to add some more context and explanations about why certain steps are needed. Also a minor note on referencing Kedro class names, when talking about a class in Kedro e.g. DataCatalog the best practice would be to use the exact class name or the regular english name for it e.g. data catalog, but not a hybrid like Data Catalog.

docs/source/deployment/airflow_astronomer.md

merelcht · 2024-04-09T08:50:33Z

docs/source/deployment/airflow_astronomer.md


-2. Create a new Kedro project using the `astro-airflow-iris` starter. You can use the default value in the project creation process:
+1. To create a new Kedro project, select the `example=yes` option to include example code. Additionally, to implement custom logging, select `tools=log`. Proceed with the default project name, but feel free to add any other tools as desired:


I think this requires a bit more of an intro to explain why the user needs the select the example project and the custom logging. Perhaps instead of immediately starting with the steps, give a bit of an intro explaining what will happen in the subsequent steps and what the expected end result is: e.g. a Kedro spaceflights project with logging setup that can be run on airflow.

docs/source/deployment/airflow_astronomer.md

merelcht · 2024-04-09T10:24:53Z

docs/source/deployment/airflow_astronomer.md

-* **Step 2.3**: Modify the `Dockerfile` to have the following content:
+This step should produce a .py file called `new_kedro_project_dag.py` located at `dags/`.
+
+### Deployment process with Astro Airflow


I would again add a bit of an introduction paragraph before diving into the steps to explain what we'll be doing in this part of the tutorial.

merelcht · 2024-04-09T10:26:07Z

docs/source/deployment/airflow_astronomer.md

+    astro dev init
+    ```
+
+2. The folder `kedro-airflow-spaceflights` will be executed within the Airflow container. To run our Kedro project there, we need to copy several items from the previous sections into it: the `/data` folder from Step 1, the `/conf` folder from Steps 2-4, the `.whl` file from Step 5, and the Airflow DAG from Step 6:


I think it would be worth explaining why these items need to be copied over, so that users will have a better understanding of how things work when they're running a different project than spaceflights on airflow.

docs/source/deployment/airflow_astronomer.md

Co-authored-by: Merel Theisen <[email protected]> Signed-off-by: Dmitry Sorokin <[email protected]>

Signed-off-by: Dmitry Sorokin <[email protected]>

DimedS · 2024-04-09T15:28:24Z

This is a fantastic start @DimedS ⭐ I have left some comments inline, but in general I think it would be useful to add some more context and explanations about why certain steps are needed. Also a minor note on referencing Kedro class names, when talking about a class in Kedro e.g. DataCatalog the best practice would be to use the exact class name or the regular english name for it e.g. data catalog, but not a hybrid like Data Catalog.

Thank you, @merelcht ! I agree and hope I've addressed your comments.

docs/source/deployment/airflow_astronomer.md

merelcht · 2024-04-10T08:47:42Z

docs/source/deployment/airflow_astronomer.md

+
+In this section, we'll start by setting up a new blank Airflow project using Astro. We'll then copy the files prepared in the previous section from our Kedro project. Next, we'll customize the Dockerfile to enhance logging capabilities and manage the installation of our Kedro package. Finally, we will run and explore the Airflow cluster.
+
+1. [Initialise an Airflow project with Astro](https://docs.astronomer.io/astro/cli/develop-project) in a new folder outside of your Kedro project. Let's call it `kedro-airflow-spaceflights`


I followed through the tutorial and got stuck at this step because I didn't have the astro CLI installed. It would be good to mention it needs to be installed.

@merelcht , we have a section Prerequisites, where that is mentioned, do you think that it's not enough? Also when you will click the link and try to install Astro CLI it will be mentioned there that you need Docker for that:

Prerequisites
To follow this tutorial, ensure you have the following:

The Astro CLI installed

Ah yes I read over the Astro CLI part. I think it won't hurt to mention again at this point in the tutorial you need the Astro CLI and Docker. I would personally also mention Docker explicitly in the prereqs.

Agreed, I've included an additional note about Docker and the Astro CLI at the start of section 2 and also mentioned Docker in the prerequisites.

merelcht · 2024-04-10T08:48:22Z

docs/source/deployment/airflow_astronomer.md

+    cp new-kedro-project/dags/new_kedro_project_dag.py kedro-airflow-spaceflights/dags/
+    ```
+
+3. Add a few lines to the `Dockerfile` to set the environment variable `KEDRO_LOGGING_CONFIG` to point to `conf/logging.yml` to enable custom logging in Kedro and to install the .whl file of our prepared Kedro project into the Airflow container:


You should specify which Dockerfile you mean, because at this point we've created new-kedro-project and kedro-airflow-spaceflights

Agree, fixed

merelcht · 2024-04-10T08:50:31Z

docs/source/deployment/airflow_astronomer.md


 RUN pip install --user dist/new_kedro_project-0.1-py3-none-any.whl
 ```

-#### Step 3. Convert the Kedro pipeline into an Airflow DAG with `kedro airflow`
+4. Navigate to `kedro-airflow-spaceflights` folder and launch the local Airflow cluster with Astronomer


I got stuck here as well, because I didn't have docker installed. So again it would be good to give some guidance or mention beforehand that users need to have astro and docker installed.

merelcht · 2024-04-10T08:59:43Z

docs/source/deployment/airflow_astronomer.md


 RUN pip install --user dist/new_kedro_project-0.1-py3-none-any.whl
 ```

-#### Step 3. Convert the Kedro pipeline into an Airflow DAG with `kedro airflow`
+4. Navigate to `kedro-airflow-spaceflights` folder and launch the local Airflow cluster with Astronomer


I got stuck here as well, because I didn't have docker installed. So again it would be good to give some guidance or mention beforehand that users need to have astro and docker installed.

Co-authored-by: Merel Theisen <[email protected]> Signed-off-by: Dmitry Sorokin <[email protected]>

Co-authored-by: Juan Luis Cano Rodríguez <[email protected]> Signed-off-by: Dmitry Sorokin <[email protected]>

Signed-off-by: Dmitry Sorokin <[email protected]>

…manual

astrojuanlu

Did a more thorough review of the prose and the workflow. Fantastic job @DimedS ! Left a few comments

astrojuanlu · 2024-04-12T08:32:06Z

docs/source/deployment/airflow.md

@@ -0,0 +1,181 @@
+# Apache Airflow
+
+Apache Airflow is a popular open-source workflow management platform. It is a suitable engine to orchestrate and execute a pipeline authored with Kedro because workflows in Airflow are modelled and organised as [DAGs](https://en.wikipedia.org/wiki/Directed_acyclic_graph).


Note for the future: we should have an entry for DAGs in our Glossary! https://docs.kedro.org/en/stable/resources/glossary.html

docs/source/deployment/airflow.md

astrojuanlu · 2024-04-12T08:38:43Z

docs/source/deployment/airflow.md

+    astro dev init
+    ```
+
+2. The folder `kedro-airflow-spaceflights` will be executed within the Airflow container. To run our Kedro project there, we need to copy several items from the previous section into it:


Suggested change

2. The folder `kedro-airflow-spaceflights` will be executed within the Airflow container. To run our Kedro project there, we need to copy several items from the previous section into it:

2. The folder `kedro-airflow-spaceflights` will be executed within the Airflow container. To run the Kedro project there, you need to copy several items from the previous section into it:

I'm wondering if there's a simpler workflow that doesn't involve copying files around. Asking because, during development, I understand that the user would want to update the configuration several times, and this involves repeating the generating of the wheel and the copying process every time.

Yes, that's a really good question. Previously, the manual described fully moving the Kedro project into the Astro project, causing them to share the same folder. This seems a bit risky to me because the two projects begin to share the same files, such as the README, requirements, and .gitignore. And it wasn't clear exactly what needed to be transferred from the Kedro project to enable it to run in Astro, so I clarified that. However, it makes sense to keep them together for the reasons you mentioned—it will be easier to work with, update Kedro, repackage, and rerun Astro from the same folder. But in this case, we should be cautious about managing these common files.

I believe it's best to keep the manual as it is to clearly illustrate how Kedro and Astro are integrated, enhancing a user's understanding. However, we could add a recommendation at the end: "Feel free to combine them in the same folder if your project requires frequent updates, DAG recreation, and repackaging." What do you think?

Sounds good!

astrojuanlu · 2024-04-12T08:40:53Z

docs/source/deployment/airflow.md

+
+## How to run a Kedro pipeline on Apache Airflow using a Kubernetes cluster
+
+The `kedro-airflow-k8s` plugin from GetInData | Part of Xebia enables you to run a Kedro pipeline on Airflow with a Kubernetes cluster. The plugin can be used together with `kedro-docker` to prepare a docker image for pipeline execution. At present, the plugin is available for versions of Kedro < 0.18 only.


cc @marrrcin @Lasica

Yeah, I think that GiD was going to deprecate this plugin as the same could be already achieved (not as easily unfortunately) with the official plugin. @Lasica

Let's continue the conversation in kedro-org/kedro-plugins#652 then, to see to what extent the official plugin can bridge the gap.

Co-authored-by: Juan Luis Cano Rodríguez <[email protected]> Signed-off-by: Dmitry Sorokin <[email protected]>

Signed-off-by: Dmitry Sorokin <[email protected]>

…manual

DimedS · 2024-04-15T08:18:49Z

I believe I've addressed most of the comments. Could you please do a final check, @ankatiyar , @merelcht , @astrojuanlu ?

astrojuanlu · 2024-04-15T13:37:50Z

@sbrugman By any chance do you have a moment to give this a look?

docs/source/deployment/airflow.md

astrojuanlu

Left a couple of nitpicks but other than that this looks good to me!

merelcht

Left a small nitpicks as well, but other than that very happy with these docs! Great work @DimedS 👍

Considering this is quite a substantial re-write, I'd also add it to the release notes.

docs/source/deployment/airflow.md

ankatiyar

LGTM, the default dataset factory is the only thing that needs to be updated!

sbrugman · 2024-04-16T07:09:12Z

docs/source/deployment/airflow.md

+Apache Airflow is a popular open-source workflow management platform. It is a suitable engine to orchestrate and execute a pipeline authored with Kedro because workflows in Airflow are modelled and organised as [DAGs](https://en.wikipedia.org/wiki/Directed_acyclic_graph).
+
+## How to run a Kedro pipeline on Apache Airflow with Astronomer
+


For users who wish to deploy on using Atronomer this tutorial is clear. For users that have a different Airflow provider, it should be mentioned which steps are not relevant and that Kedro Airflow is not specific to astronomer.

In the logical flow (reading top to bottom), this information should be placed above Astronomer specific instructions

ankatiyar

Was following along the tutorial and came across v minor things

docs/source/deployment/airflow.md

…manual

Co-authored-by: Merel Theisen <[email protected]> Co-authored-by: Juan Luis Cano Rodríguez <[email protected]> Signed-off-by: Ankita Katiyar <[email protected]>

Signed-off-by: Ankita Katiyar <[email protected]>

…manual

Update Airflow Astro deployment docs

41fc47d

Signed-off-by: Dmitry Sorokin <[email protected]>

DimedS linked an issue Apr 8, 2024 that may be closed by this pull request

kedro-airflow: revise Airflow deployment manual kedro-org/kedro-plugins#605

Closed

DimedS requested a review from ankatiyar April 8, 2024 15:50

DimedS and others added 2 commits April 8, 2024 18:54

Merge branch 'main' into 605-kedro-airflow-revise-airflow-deployment-…

02916f5

…manual

Linter

0ace5a3

Signed-off-by: Dmitry Sorokin <[email protected]>

merelcht reviewed Apr 9, 2024

View reviewed changes

DimedS and others added 9 commits April 9, 2024 11:34

Update docs/source/deployment/airflow_astronomer.md

73ea578

Co-authored-by: Merel Theisen <[email protected]> Signed-off-by: Dmitry Sorokin <[email protected]>

Update docs/source/deployment/airflow_astronomer.md

39ba979

Co-authored-by: Merel Theisen <[email protected]> Signed-off-by: Dmitry Sorokin <[email protected]>

Update docs/source/deployment/airflow_astronomer.md

c7081a2

Co-authored-by: Merel Theisen <[email protected]> Signed-off-by: Dmitry Sorokin <[email protected]>

Update docs/source/deployment/airflow_astronomer.md

4694c9f

Co-authored-by: Merel Theisen <[email protected]> Signed-off-by: Dmitry Sorokin <[email protected]>

Update docs/source/deployment/airflow_astronomer.md

95b1875

Co-authored-by: Merel Theisen <[email protected]> Signed-off-by: Dmitry Sorokin <[email protected]>

Update docs/source/deployment/airflow_astronomer.md

3e30192

Co-authored-by: Merel Theisen <[email protected]> Signed-off-by: Dmitry Sorokin <[email protected]>

Update docs/source/deployment/airflow_astronomer.md

fc8de8f

Co-authored-by: Merel Theisen <[email protected]> Signed-off-by: Dmitry Sorokin <[email protected]>

Add sections description

075c443

Signed-off-by: Dmitry Sorokin <[email protected]>

Linter

44a41a3

Signed-off-by: Dmitry Sorokin <[email protected]>

DimedS requested a review from merelcht April 9, 2024 15:28

astrojuanlu reviewed Apr 10, 2024

View reviewed changes

docs/source/deployment/airflow_astronomer.md Outdated Show resolved Hide resolved

merelcht reviewed Apr 10, 2024

View reviewed changes

DimedS and others added 7 commits April 10, 2024 11:29

Apply suggestions from code review

f6fa121

Co-authored-by: Merel Theisen <[email protected]> Signed-off-by: Dmitry Sorokin <[email protected]>

Update docs/source/deployment/airflow_astronomer.md

cccde67

Co-authored-by: Juan Luis Cano Rodríguez <[email protected]> Signed-off-by: Dmitry Sorokin <[email protected]>

Add comments

c1ea526

Signed-off-by: Dmitry Sorokin <[email protected]>

Add Cloud section

e1c2f25

Signed-off-by: Dmitry Sorokin <[email protected]>

Update header

190470b

Signed-off-by: Dmitry Sorokin <[email protected]>

Rename file

ecdd6f3

Signed-off-by: Dmitry Sorokin <[email protected]>

Merge branch 'main' into 605-kedro-airflow-revise-airflow-deployment-…

3a8afd9

…manual

DimedS marked this pull request as ready for review April 11, 2024 11:52

DimedS requested a review from yetudada as a code owner April 11, 2024 11:52

DimedS requested review from astrojuanlu and merelcht April 11, 2024 11:54

astrojuanlu reviewed Apr 12, 2024

View reviewed changes

DimedS and others added 6 commits April 12, 2024 10:14

Apply suggestions from code review

772f60c

Co-authored-by: Juan Luis Cano Rodríguez <[email protected]> Signed-off-by: Dmitry Sorokin <[email protected]>

Fix link

7d4fc7f

Signed-off-by: Dmitry Sorokin <[email protected]>

Fix link

a0768a0

Signed-off-by: Dmitry Sorokin <[email protected]>

Dataset Factories

e8daf96

Signed-off-by: Dmitry Sorokin <[email protected]>

Add group-in-memory

2d508a0

Signed-off-by: Dmitry Sorokin <[email protected]>

Merge branch 'main' into 605-kedro-airflow-revise-airflow-deployment-…

420220f

…manual

DimedS requested review from astrojuanlu and ankatiyar April 15, 2024 08:18

DimedS mentioned this pull request Apr 15, 2024

kedro-airflow: revise Airflow deployment manual kedro-org/kedro-plugins#605

Closed

astrojuanlu requested a review from sbrugman April 15, 2024 13:37

astrojuanlu reviewed Apr 15, 2024

View reviewed changes

docs/source/deployment/airflow.md Outdated Show resolved Hide resolved

astrojuanlu reviewed Apr 15, 2024

View reviewed changes

docs/source/deployment/airflow.md Outdated Show resolved Hide resolved

astrojuanlu reviewed Apr 15, 2024

View reviewed changes

docs/source/deployment/airflow.md Outdated Show resolved Hide resolved

astrojuanlu approved these changes Apr 15, 2024

View reviewed changes

merelcht approved these changes Apr 15, 2024

View reviewed changes

docs/source/deployment/airflow.md Outdated Show resolved Hide resolved

docs/source/deployment/airflow.md Outdated Show resolved Hide resolved

docs/source/deployment/airflow.md Outdated Show resolved Hide resolved

ankatiyar approved these changes Apr 15, 2024

View reviewed changes

sbrugman reviewed Apr 16, 2024

View reviewed changes

ankatiyar reviewed Apr 16, 2024

View reviewed changes

docs/source/deployment/airflow.md Outdated Show resolved Hide resolved

docs/source/deployment/airflow.md Show resolved Hide resolved

ankatiyar and others added 4 commits April 19, 2024 12:12

Merge branch 'main' into 605-kedro-airflow-revise-airflow-deployment-…

49a184a

…manual

Apply suggestions from code review

a001ccd

Co-authored-by: Merel Theisen <[email protected]> Co-authored-by: Juan Luis Cano Rodríguez <[email protected]> Signed-off-by: Ankita Katiyar <[email protected]>

Fix the factory dataset, numbering and add to release notes

fcfbf2b

Signed-off-by: Ankita Katiyar <[email protected]>

Merge branch 'main' into 605-kedro-airflow-revise-airflow-deployment-…

292c954

…manual

ankatiyar enabled auto-merge (squash) April 19, 2024 11:49

ankatiyar merged commit 144323a into main Apr 19, 2024
10 checks passed

ankatiyar deleted the 605-kedro-airflow-revise-airflow-deployment-manual branch April 19, 2024 11:58

This was referenced May 24, 2024

fix(airflow): fix the link to correct deployment manual kedro-org/kedro-plugins#697

Merged

Kedro-Airflow not working with Astrocloud kedro-org/kedro-plugins#13

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Airflow Astro deployment docs #3792

Update Airflow Astro deployment docs #3792

DimedS commented Apr 8, 2024

merelcht left a comment

merelcht Apr 9, 2024

merelcht Apr 9, 2024

merelcht Apr 9, 2024

DimedS commented Apr 9, 2024

merelcht Apr 10, 2024

DimedS Apr 10, 2024

merelcht Apr 10, 2024

DimedS Apr 11, 2024

merelcht Apr 10, 2024

DimedS Apr 11, 2024

merelcht Apr 10, 2024

merelcht Apr 10, 2024

astrojuanlu left a comment

astrojuanlu Apr 12, 2024

astrojuanlu Apr 12, 2024

astrojuanlu Apr 12, 2024

DimedS Apr 12, 2024

DimedS Apr 12, 2024

astrojuanlu Apr 12, 2024

DimedS Apr 15, 2024

astrojuanlu Apr 12, 2024

marrrcin Apr 16, 2024

astrojuanlu Apr 16, 2024

DimedS commented Apr 15, 2024

astrojuanlu commented Apr 15, 2024

astrojuanlu left a comment

merelcht left a comment

ankatiyar left a comment

sbrugman Apr 16, 2024

ankatiyar left a comment


		2. Create a new Kedro project using the `astro-airflow-iris` starter. You can use the default value in the project creation process:
		1. To create a new Kedro project, select the `example=yes` option to include example code. Additionally, to implement custom logging, select `tools=log`. Proceed with the default project name, but feel free to add any other tools as desired:


		In this section, we'll start by setting up a new blank Airflow project using Astro. We'll then copy the files prepared in the previous section from our Kedro project. Next, we'll customize the Dockerfile to enhance logging capabilities and manage the installation of our Kedro package. Finally, we will run and explore the Airflow cluster.

		1. [Initialise an Airflow project with Astro](https://docs.astronomer.io/astro/cli/develop-project) in a new folder outside of your Kedro project. Let's call it `kedro-airflow-spaceflights`

		@@ -0,0 +1,181 @@
		# Apache Airflow

		Apache Airflow is a popular open-source workflow management platform. It is a suitable engine to orchestrate and execute a pipeline authored with Kedro because workflows in Airflow are modelled and organised as [DAGs](https://en.wikipedia.org/wiki/Directed_acyclic_graph).

	2. The folder `kedro-airflow-spaceflights` will be executed within the Airflow container. To run our Kedro project there, we need to copy several items from the previous section into it:
	2. The folder `kedro-airflow-spaceflights` will be executed within the Airflow container. To run the Kedro project there, you need to copy several items from the previous section into it:


		## How to run a Kedro pipeline on Apache Airflow using a Kubernetes cluster

		The `kedro-airflow-k8s` plugin from GetInData \| Part of Xebia enables you to run a Kedro pipeline on Airflow with a Kubernetes cluster. The plugin can be used together with `kedro-docker` to prepare a docker image for pipeline execution. At present, the plugin is available for versions of Kedro < 0.18 only.

Update Airflow Astro deployment docs #3792

Update Airflow Astro deployment docs #3792

Conversation

DimedS commented Apr 8, 2024

Description

Developer Certificate of Origin

Checklist

merelcht left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DimedS commented Apr 9, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

astrojuanlu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DimedS commented Apr 15, 2024

astrojuanlu commented Apr 15, 2024

astrojuanlu left a comment

Choose a reason for hiding this comment

merelcht left a comment

Choose a reason for hiding this comment

ankatiyar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ankatiyar left a comment

Choose a reason for hiding this comment