Proposal/2071 update version versioning

[RNHTTR]

Overview

This PR formally introduces Proposal: Update DatasetVersion versioning #2071

Checklist

• You've signed-off your work
• Your changes are accompanied by tests (if relevant)
• Your change contains a small diff and is self-contained
• You've updated any relevant documentation (if relevant)
• You've updated the CHANGELOG.md with details about your change under the "Unreleased" section (if relevant, depending on the change, this may not be necessary)
• You've versioned your .sql database schema migration according to Flyway's naming convention (if relevant)
• You've included a header in any source code files (if relevant)

[RNHTTR]

I could use some help with a couple things here:

1. Confirming the proposed db changes are backward compatible
2. Added details for implementing the facet. I assume it'll be part of the DatasetVersionDAO, but I'm not entirely sure.
3. Authoring db migration

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (6eb0ee6) 83.76% compared to head (26175f9) 75.78%.

❗ Current head 26175f9 differs from pull request most recent head 8c88a32. Consider uploading reports for the commit 8c88a32 to get more accurate results

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2153      +/-   ##
============================================
- Coverage     83.76%   75.78%   -7.99%     
+ Complexity     1338     1061     -277     
============================================
  Files           247      209      -38     
  Lines          6112     5006    -1106     
  Branches        281      403     +122     
============================================
- Hits           5120     3794    -1326     
+ Misses          843      763      -80     
- Partials        149      449     +300     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[RNHTTR]

FYI @collado-mike @wslulciuc

[collado-mike]

Can you comment on how the public API is impacted (if at all)?

[RNHTTR]

Not from a user's perspective as far as I can tell, although there is another JobVersion in the public API that will need the version field removed. I'll also need to update the various versions in the clients. I'll add all of these to the list of TODOs in the implementation section.

[RNHTTR]

Actually, I guess, while the public API will be backward compatible, we'll want to expose externalVersion to the public API as well.

[RNHTTR]

Updated the proposal

[collado-mike]

I was thinking specifically about how the URLs for fetching DatasetVersions will work and how we return references to the DatasetVersion. E.g., this structure in the spec references a UUID version, but I don't know if that is going to change or if it will just reference the DatasetVersion's primary key. So what is returned in the Run spec's inputDatasetVersions field?
Additionally, the getDatasetVersion spec accepts a version parameter. Is that the UUID? or the externalVersion?
Personally, I prefer the externalVersion, if present, as it introduces compatibility between systems - e.g., a Snowflake dashboard can just know how to construct a link to the Marquez UI for a given dataset@version because the standard is known and consistent.

[wslulciuc]

@collado-mike: very good points on backwards compatibility. To ensure we don't introduce any breaking changes, Marquez will continue to reference the uuid of dataset_versions and set that value accordingly. That is, in the case of Datasets.currentVersion, the value will continue to be of type uuid, but with the addition of Datasets.currentExternalVersion (we can think of this as an alias). More specifically, for DatasetVersionId, we will add Datasets.externalVersion, which can be null, but Datasets.version will always be set.

I think retrieving a dataset version is less obvious, but we can update the endpoint:

GET /namespaces/{namespace}/datasets/{dataset}/versions/{version}

where {version} can either be a uuid or a string type. The parsing on the backend really isn't that complicated and be easy enough to do.

Another option would be we expose a new endpoint, but that seems unnecessary and would prefer being flexible on {version} type:

GET /namespaces/{namespace}/datasets/{dataset}/versions/{version}/external

/cc @RNHTTR

[wslulciuc]

Great proposal, @RNHTTR! I'm going to merge as this PR has been around for a while (really my fault here 😅). I've responded with my thoughts on backwards compatibility, we can follow up, but feel what you outlined for versioning is in a state where someone can pick up the work.

[netlify]

✅ Deploy Preview for peppy-sprite-186812 canceled.

Name	Link
🔨 Latest commit	8c88a32
🔍 Latest deploy log	https://app.netlify.com/sites/peppy-sprite-186812/deploys/6553b618586262000883faeb