Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK] support column lineage for merge into on databricks #2348

Merged
merged 1 commit into from
Jan 3, 2024

Conversation

pawel-big-lebowski
Copy link
Collaborator

@pawel-big-lebowski pawel-big-lebowski commented Jan 2, 2024

Problem

merge into is not supported on Databricks platform which is a result of MergeIntoCommandEdge node within logical plan, where MergeIntoCommandEdge is not present within Spark nor Delta codebase.

Solution

Extract lineage from MergeIntoCommandEdge class.

Note: All schema changes require discussion. Please link the issue for context.

  • Your change modifies the core OpenLineage model
  • Your change modifies one or more OpenLineage facets

If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports S3 and GCS filesystem operations, tested with AWS EMR).

One-line summary:

Checklist

  • You've signed-off your work
  • Your pull request title follows our guidelines
  • Your changes are accompanied by tests (if relevant)
  • Your change contains a small diff and is self-contained
  • You've updated any relevant documentation (if relevant)
  • Your comment includes a one-liner for the changelog about the specific purpose of the change (if necessary)
  • You've versioned the core OpenLineage model or facets according to SchemaVer (if relevant)
  • You've added a header to source files (if relevant)

SPDX-License-Identifier: Apache-2.0
Copyright 2018-2023 contributors to the OpenLineage project

@pawel-big-lebowski pawel-big-lebowski force-pushed the spark/databricks-merge-into branch 5 times, most recently from ba42dfd to 6424cd1 Compare January 3, 2024 07:47
@boring-cyborg boring-cyborg bot added the area:documentation Improvements or additions to documentation label Jan 3, 2024
@pawel-big-lebowski pawel-big-lebowski marked this pull request as ready for review January 3, 2024 10:30
@pawel-big-lebowski
Copy link
Collaborator Author

@pawel-big-lebowski pawel-big-lebowski merged commit 340d1b6 into main Jan 3, 2024
22 checks passed
@pawel-big-lebowski pawel-big-lebowski deleted the spark/databricks-merge-into branch January 3, 2024 14:37
fafnirZ pushed a commit to fafnirZ/OpenLineage that referenced this pull request Jul 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:documentation Improvements or additions to documentation area:integration/spark
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants