-
Notifications
You must be signed in to change notification settings - Fork 14.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for openlineage to AFS and common.io #36410
Conversation
should I just add openlineage to the list to 'always' test @potiuk ? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good, I just added a comment about the change on key extraction.
Pulling @mobuchowski to the conversation since he mentioned he wants to investigate adding this some time ago. |
@mobuchowski if there is any way to emit lineage events and openlineage can reconcile those we can add those events at the lower level so lineage becomes available for TaskFlow as well without manual intervention. Not sure if we can though? |
@bolkedebruin working on it 🙂 f15a1e0 |
This adds low level support for open lineage to ObjectStorage and integrates it into common.io.
2cf89a0
to
563cc5e
Compare
ptal @hussein-awala @potiuk @uranusjr. The doc caching update has been split from this pr, I've kept the resolution of a |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good
This adds low level support for open lineage to ObjectStorage and integrates it into common.io. (cherry picked from commit 33996a4)
This adds low level support for open lineage to ObjectStorage and integrates it into common.io.
This adds low level support for open lineage to ObjectStorage and integrates it into common.io.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rst
or{issue_number}.significant.rst
, in newsfragments.