feat: Add ManagedTableDataset for managed Delta Lake tables in Databricks

[jmholzer]

Description

A fork PR of #127, close #127

Development notes

Checklist

• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the relevant RELEASE.md file
• Added tests to cover my changes

[noklam]

Thanks for the PR. I cannot manually test it so I mainly focus on reviewing the code itself.

One additional question I have is that this seem to inherit from AbstractVersionedDataSet, how does the versioning work since it's a Table if I understand correctly, does it works with the Kedro versioning?

[noklam]

Why do pandas dependency matter here? I expect Pyspark has it own pandas dependency pin already (if needed)

[jmholzer]

Pandas >= 2 removed a method that that Spark < 3.4.0 relies on. This was fixed in Spark 3.4.0. With Spark < 3.4.0 and Pandas >= 2, new tests fail.

There are two options for getting tests to pass here:

1. require PySpark >= 3.4.0, or
2. require Pandas < 2.0

I went for option 2, as pinning PySpark ~= 3.4.0 (and therefore Spark 3.4.0) causes a dependency issue with delta-spark and causes all new tests to fail (plus Spark tests to fail as soon as the import statements in our test files are executed), whereas requiring Pandas < 2.0 caused no other problems with the test suite. Since there is some pressure to get this released and getting our tests to work with Spark 3.4.0 will take some work, I went for option 2.

I would keep it like this, then open an issue to upgrade our Spark to 3.4.0 asap. I can tackle this issue after the Databricks work is delivered.

[jmholzer]

One additional question I have is that this seem to inherit from AbstractVersionedDataSet, how does the versioning work since it's a Table if I understand correctly, does it works with the Kedro versioning?

Good question. When delta lake files are modified in any way, a new version of the table is created. Versions of a delta lake table can be accessed either a version number (ascending enum).

ManagedTableDataset currently handles versioning by allowing the user to pass in an object of kedro.io.core.Version specifying this version number. When the table is modified (by any of the _save methods), a new table version is automatically saved in the delta lake file itself. Since versioning is handled by the delta lake format, we do not have to worry about it ourselves.

To my understanding, this is compatible with theAbstractVersionedDataset interface, what do you think?

[noklam]

thanks for the work!

I think the versioning is fine, is it specific to databricks or any delta table support versioning? (sorry I may get confused with all the delta terminology?) If this is something supported with delta table then we may even extend it to some datasets like SQLtabledataset.

The versioning feature relies on a file path and doesn't work for tables. It relies on the timestamp so basically this dataset implement it's own logic to fetch latest version of dataset. kedro-org/kedro#1778 we should consider this when we start this piece of work.

[jmholzer]

Thanks for the quick review! Appreciate it 🙂.

I think the versioning is fine, is it specific to databricks or any delta table support versioning? (sorry I may get confused with all the delta terminology?)

All delta lake files support versioning in this way, regardless of whether they are saved via DBFS or not.

[SajidAlamQB]

This looks great, just some minor comments. 🌟

[SajidAlamQB]

These look to be doing the same thing, could we not just take in an argument to determine if it's a table, database or catalog and have 1 method?

[jmholzer]

This is a great push. Managed tables are stored in the hierarchy Catalog -> Database -> Table. A given table needs at least a database name and a table name specified, as a catalog name can (I think) be inferred.

I thought about it a bit in response to your comment, and I added a step that also raises a DatasetError if no database name or no table name is specified. With these changes it is only validate_catalog that has a different structure to the other two namespace validation methods. This being the case, my view is that is clearer to keep these two methods separate rather than DRY them.