fix column lineage returning multiple entries for job run multiple times

[pawel-big-lebowski]

Signed-off-by: Pawel Leszczynski [email protected]

Problem

Each column lineage row contains columns: output_dataset_version_uuid, output_dataset_field_uuid, input_dataset_version_uuid, input_dataset_field_uuid. This means that in case of a single job reading one column and writing to another, if a job is run twice this will result in two rows in column-lineage table. This is fine as we want to track this information. But we shouldn't retrieve this twice through the endpoint.

Solution

Return column dependency only once if a job has been run several times.

Note: All database schema changes require discussion. Please link the issue for context.

Checklist

• You've signed-off your work
• Your changes are accompanied by tests (if relevant)
• Your change contains a small diff and is self-contained
• You've updated any relevant documentation (if relevant)
• You've updated the CHANGELOG.md with details about your change under the "Unreleased" section (if relevant, depending on the change, this may not be necessary)
• You've versioned your .sql database schema migration according to Flyway's naming convention (if relevant)
• You've included a header in any source code files (if relevant)

[codecov]

Codecov Report

Merging #2176 (5e496ec) into main (496566e) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##               main    #2176   +/-   ##
=========================================
  Coverage     76.33%   76.33%           
  Complexity     1099     1099           
=========================================
  Files           214      214           
  Lines          5139     5139           
  Branches        407      407           
=========================================
  Hits           3923     3923           
  Misses          762      762           
  Partials        454      454           

Impacted Files	Coverage Δ
api/src/main/java/marquez/db/ColumnLineageDao.java	100.00% <ø> (ø)

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more