Column lineage graph endpoint

[pawel-big-lebowski]

Signed-off-by: Pawel Leszczynski [email protected]

Problem

PR #2096 allows storing in database column-lineage information from the events. In this PR we expose column lineage through a graph endpoint according to the proposal (https://github.com/MarquezProject/marquez/blob/main/proposals/2045-column-lineage-endpoint.md)

Closes: #2114

Solution

• Another NodeType DATASET_FIELD is added,
• column-lineage endpoint returns serialized Lineage objects similar to a currently existing lineage endpoint,

Note: All database schema changes require discussion. Please link the issue for context.

Checklist

• You've signed-off your work
• Your changes are accompanied by tests (if relevant)
• [] Your change contains a small diff and is self-contained
• You've updated any relevant documentation (if relevant)
• You've updated the CHANGELOG.md with details about your change under the "Unreleased" section (if relevant, depending on the change, this may not be necessary)
• You've versioned your .sql database schema migration according to Flyway's naming convention (if relevant)
• You've included a header in any source code files (if relevant)

[codecov]

Codecov Report

Merging #2124 (8e66689) into main (b9abb19) will increase coverage by 0.51%.
The diff coverage is 91.83%.

@@             Coverage Diff              @@
##               main    #2124      +/-   ##
============================================
+ Coverage     75.82%   76.33%   +0.51%     
- Complexity     1063     1099      +36     
============================================
  Files           209      214       +5     
  Lines          5013     5139     +126     
  Branches        403      407       +4     
============================================
+ Hits           3801     3923     +122     
+ Misses          763      762       -1     
- Partials        449      454       +5     

Impacted Files	Coverage Δ
api/src/main/java/marquez/db/ColumnLineageDao.java	100.00% <ø> (ø)
api/src/main/java/marquez/db/DatasetFieldDao.java	100.00% <ø> (ø)
.../src/main/java/marquez/service/DelegatingDaos.java	0.00% <ø> (ø)
...main/java/marquez/service/models/LineageEvent.java	83.56% <75.00%> (-1.52%)	⬇️
...c/main/java/marquez/api/ColumnLineageResource.java	80.00% <80.00%> (ø)
...i/src/main/java/marquez/service/models/NodeId.java	64.48% <80.00%> (+2.38%)	⬆️
api/src/main/java/marquez/db/OpenLineageDao.java	95.53% <86.66%> (+0.24%)	⬆️
...arquez/db/mappers/ColumnLineageNodeDataMapper.java	90.00% <90.00%> (ø)
...ain/java/marquez/service/ColumnLineageService.java	97.14% <97.14%> (ø)
api/src/main/java/marquez/MarquezContext.java	85.71% <100.00%> (+0.78%)	⬆️
... and 7 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[pawel-big-lebowski]

Most important piece of the PR: recursive query to extract column-lineage graph.
Only column_lineage table is used and joined to obtained graph nodes.
Other tables are only used to enrich found nodes.

[pawel-big-lebowski]

Fix to previous PR to align with Openlineage spec:
https://github.com/OpenLineage/OpenLineage/blob/main/spec/facets/ColumnLineageDatasetFacet.json