feat: kedro-airflow group in memory nodes

[sbrugman]

Description

When running Kedro nodes using Airflow, MemoryDataSets are often not shared across operators. This will cause the DAG run to fail.

MemoryDataSets may be used to provide logical separation between nodes in Kedro, without the overhead of needing to write to disk (and in the case of distributed running needing multiple executors).

This PR adds an option to group all nodes that are connected through MemoryDataSets together. This preserves the option to have logical separation in Kedro, with little computational overhead.

(Previously raised in kedro-k8s-airflow)

Development notes

Built on top of #233, as we use the Kedro session. I can make them independent if there are blockers for that PR.

Checklist

• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the relevant RELEASE.md file
• Added tests to cover my changes

[sbrugman]

Blocked by #233

[astrojuanlu]

After #233 was merged, @sbrugman do you still want to pursue this PR?

[sbrugman]

Yes, I'll rebase this one!

[sbrugman]

Fyi still on my list to finish this feature, preferably when #335 is merged

[marrrcin]

FYI, we have a working PoC which does the grouping based on node tags (incl. validation).

[sbrugman]

This PR would also be simple to extend to pick up other criteria then the MemoryDataSets, once #335 is merged

[astrojuanlu]

This can now be rebased 🚀

[sbrugman]

Rebased the PR; ready for review.

This implementation concatenates the names of the nodes when grouped, however this can be customized by passing a custom jinja template.

It uses the kedro run --nodes node1 --nodes n2 parameter to run multiple nodes, however one can choose to use Airflow's task groups (via the jinja template), or any other kind of grouping feature.

The data structure of the dependencies has changed in such a way that other forms of grouping (notably via tags) can also be supported by a future PR (very much welcome).

(The feature was also tested for our in-house Kedro to Airflow conversion)

[datajoely]

@sbrugman if you have any time I'd really appreciate your thoughts on this research document
kedro-org/kedro#3094

[sbrugman]

@astrojuanlu could we please merge this?
We are using this already and install from git. Would be great to use PyPI again

Ps. Congrats with the release!

[astrojuanlu]

@sbrugman let's get this over the line yes - could you fix the conflicts? And we'll have a look soon

[sbrugman]

Rebased the changes and fixed the merge conflicts.

[sbrugman]

Errors seem unrelated to the PR. The tests are outdated. kedro-airflow/features/steps/cli_steps.py refers to pandas-iris on L162

[ankatiyar]

@astrojuanlu @sbrugman I'm working on a post 0.19.x release cleanup for airflow #478. That should fix the CI issues and then we can get this in before releasing kedro-airflow

[sbrugman]

I'm fine with requiring Kedro >= 0.19 for this plugin with this feature. The backwards compatibility is most likely not worth it assuming the few (but grateful) users of the plugin

[astrojuanlu]

+1 and users stuck in Kedro<0.19 can always use the older version of the plugin.

[sbrugman]

Updated the PR with all feedback. Let's get this one through :)

[astrojuanlu]

If more work is needed, we can polish as we go :) Thanks a lot for your patience @sbrugman !