-
Notifications
You must be signed in to change notification settings - Fork 14.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Ability to add custom facet in OpenLineage events #38982
Conversation
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
|
446c159
to
435db7d
Compare
Hi @eladkal , @anandhimurali is my colleague from the same team and I'm really excited about her 1st PR to Airflow! Do you have an idea who among the committers may be the best to help review this openlineage-related change? Any suggestion/input would be greatly appreciated. Thanks a lot! |
Ping @mobuchowski for review :) |
Hey @anandhimurali @XD-DENG I think ability to add custom facets to arbitrary operators is a great idea. I have two points I think it's worth addressing:
Overall I like the solution, but might be good to specify the goals regarding 1) and 2) - because for sure, later, someone will come and ask "how to make this custom facet work only on some particular tasks" and "how do I pass some custom info to my function". |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like the idea 🚀 Apart from what @mobuchowski mentioned above, I added some small comments about the code itself.
Hey @anandhimurali @XD-DENG, is there anything i could help you with here? I really like the idea and would like to see it happen, let me know 😄 |
Hi @kacpermuda, Thanks for the interest and I'm excited to hear that. Sorry I've been busy for the last few weeks. I'll look into the comments this week. |
d0265d1
to
b4414cb
Compare
Hi @mobuchowski,
Let me know if this needs to be documented anywhere. Thanks. |
6e96d73
to
03ed172
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
421e0fe
to
e851940
Compare
@anandhimurali any interest in getting this PR into shape and approved? I think this is good, really useful feature. |
@mobuchowski Yes, definitely. I just have to add some documentation and will get this done this week. Thanks and sorry for the delay. |
4f193aa
to
0d63f59
Compare
I think it would be good to rename this functionality to This is a great feature:) |
Co-authored-by: Kacper Muda <[email protected]>
Co-authored-by: Kacper Muda <[email protected]>
81a8282
to
8c404b9
Compare
tests/system/providers/google/cloud/datapipelines/resources/__init__.py
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
one small thing, otherwise great job! 🚀
Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions. |
Thanks @anandhimurali for implementing this feature 🚀 |
* Ability to add custom facet in OpenLineage events * Update airflow/providers/openlineage/provider.yaml Co-authored-by: Elad Kalif <[email protected]> * Update airflow/providers/openlineage/provider.yaml Co-authored-by: Kacper Muda <[email protected]> * Adding None type hint for the custom facet function * Fix a test after rebase * Removed the legacy OPENLINEAGE_ configs format for OPENLINEAGE_CUSTOM_FACET_FUNCTIONS * Duplicate facet key check * Update airflow/providers/openlineage/utils/utils.py Co-authored-by: Kacper Muda <[email protected]> * Update airflow/providers/openlineage/utils/utils.py Co-authored-by: Kacper Muda <[email protected]> * Fixes after rebase * Adding user docs for custom_facet_functions * Rename custom_facet_functions as custom_run_facets * Increment version for custom_run_facets feature * Enrich example with access to operator and return value as None. * Add try-except for custom facet function execution * Fix the typing for the custom facet fucntion return type * Documentation: funcs are executed only for START events * Fix the typing for the custom facet function return type * Fixes after pre-commit hook checks * Adding start_date to test DAGs for 2.7 compatibility tests * Removing a out of scope __init__ file added by pre-commit check --------- Co-authored-by: Anandhi <[email protected]> Co-authored-by: Elad Kalif <[email protected]> Co-authored-by: Kacper Muda <[email protected]>
Motivation
Currently, users need to write custom extractor inorder to add custom facets in the lineage events. Instead, providing a way to inject custom facets by accepting a config.
Modifications
The
get_custom_facets
function in the OpenLineage provider is enhanced to fetch the Airflow configAIRFLOW__OPENLINEAGE__CUSTOM_RUN_FACETS
, execute those functions and append their return values to the run facet of the lineage event for all the operators.Result
Enables users to add custom facets to the lineage events by defining functions instead of custom extractor.
Configs documentation for
AIRFLOW__OPENLINEAGE__CUSTOM_RUN_FACETS
User Docs
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rst
or{issue_number}.significant.rst
, in newsfragments.