add StorageDatasetFacet to spec

[pawel-big-lebowski]

Signed-off-by: Pawel Leszczynski [email protected]

Problem

Spark integration collects table provider information like delta or iceberg within custom facet. This information becomes useful together with a dataset version collected as version is retrieved from the provider. To make it consistent, if a version is within a global spec, the provider information should also be put there, however a more generic name should be considered like storageProvider, backendStorageProvider, storageProperties with fields: format or storageProvider

Closes: #619

Solution

Add datasetProvider facet to spec.

Note: All schema changes require discussion. Please link the issue for context.

• Your change modifies the core OpenLineage model
• Your change modifies one or more OpenLineage facets

If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports S3 and GCS filesystem operations, tested with AWS EMR).

Checklist

• [] You've signed-off your work
• [] Your pull request title follows our guidelines
• Your changes are accompanied by tests (if relevant)
• Your change contains a small diff and is self-contained
• You've updated any relevant documentation (if relevant)
• You've updated the CHANGELOG.md with details about your change under the "Unreleased" section (if relevant, depending on the change, this may not be necessary)
• You've versioned the core OpenLineage model or facets according to SchemaVer (if relevant)

[julienledem]

Thanks for starting this

[julienledem]

Since we are already in the "provider" facet, this would be "name".
We should document a list of known providers to ensure consistency. Would that apply to other things than Table Formats? Possibly a name more specific than "provider" would be appropriate.

[pawel-big-lebowski]

Given the discussion we have, term provider seems to be kind of misleading.
Official iceberg and delta definitions are:

• Iceberg is a high-performance format for huge analytic tables
• Delta Lake is a storage layer that brings scalable, ACID transactions to Apache Spark and other big-data engines.

Following that, we could name it storage of the type DatasetStorageDatasetFacet with the fields:

• storageLayer which can be iceberg, delta
• fileFormat which can be parquet, orc, etc.

Alternatively we could go with StorageLayerDatasetFacet which is also OK but has a limited extensibility in the future.

[pawel-big-lebowski]

I modified facet into DatasetStorageDatasetFacet as it describes better its content.

[julienledem]

Thank you.
The name of the field (here "storage") should be consistent with the name of the type so that makes it "StorageDatasetFacet"

[pawel-big-lebowski]

Thanks for the comments. I've applied the changes accordingly.

[julienledem]

Similar comment: we should document known formats

[mobuchowski]

Add to https://github.com/OpenLineage/OpenLineage/blob/main/spec/Naming.md perhaps?

[julienledem]

I think the description here should be more directive:
It is "iceberg" or "delta"
not "like iceberg or delta"

[julienledem]

I made few last comments. Otherwise this looks great to me.