feat(datasets): Added DeltaTableDataSet

[afaqueahmad7117]

Description

This PR adds DeltaTableDataSet to the Kedro datasets plugin. Issue 226.

Development notes

1. DeltaTableDataSet here kedro-datasets/kedro_datasets/pandas/deltatable_dataset.py
2. Unit Tests kedro-datasets/tests/pandas/test_deltatable_dataset.py

Checklist

• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the relevant RELEASE.md file
• Added tests to cover my changes

[everdark]

Just a side note, there is a bug for the delta-rs python binding for using catalog: delta-io/delta-rs#1466; so right now I'm not able to smoke test the from_data_catalog method to see if it actually works.

[noklam]

@everdark Is this ready to be review now? I saw you have closed the old PR but this one is still in draft.

[everdark]

@everdark Is this ready to be review now? I saw you have closed the old PR but this one is still in draft.

Depending on whether @afaqueahmad7117 is doing some more works. I don't have anything to add from my side as of now. We close the old PR cause there are some mess-up in the commits.

[afaqueahmad7117]

@noklam @everdark I'm done with my changes. Marked PR ready for review.

[noklam]

Noted that the CI is failing for some other reason, kedro-org/kedro#2673 would fix this, so ignore the irrelevant error or try to install a slightly older kedro version to test it locally.

[astrojuanlu]

The new dataset does not yet have 100 % coverage, some lines are missing:

kedro_datasets/pandas/deltatable_dataset.py                66      3    95%   153, 211, 216

[everdark]

The new dataset does not yet have 100 % coverage, some lines are missing:

kedro_datasets/pandas/deltatable_dataset.py                66      3    95%   153, 211, 216

Coverage increased to 100% in 2b1c5e8.

[astrojuanlu]

Thanks a lot folks, this is looking really good!

Just to help me contextualize and sorry for the basic question: but could you summarize what is the difference between this DeltaTableDataSet and the existing ManagedTableDataSet? I see the latter can be used with Delta tables too https://kedro.org/blog/managed-delta-tables-kedro-dataset

[everdark]

Thanks a lot folks, this is looking really good!

Just to help me contextualize and sorry for the basic question: but could you summarize what is the difference between this DeltaTableDataSet and the existing ManagedTableDataSet? I see the latter can be used with Delta tables too https://kedro.org/blog/managed-delta-tables-kedro-dataset

The ManagedTableDataSet is for delta table managed by Databricks Unity Catalog. It is a very specific (proprietary) implementation on top of the open source delta table. Our dataset is a generic non-spark solution to handle the open source version of the delta table.

By adding this into kedro-datasets, there will be 3 possible ways of handling delta table:

1. Apache Spark
2. delta-rs, a non-Spark approach (this PR)
3. Databricks Unity Catalog

The user should decide which one best suits their need.

[noklam]

By adding this into kedro-datasets, there will be 3 possible ways of handling delta table:
Apache Spark
delta-rs, a non-Spark approach (this PR)
Databricks Unity Catalog

Agree, although DeltaTable is often associated with Spark, but it's actually just a file format and you can read it via Pandas or maybe other libraries later.

I think the current space is still dominated by Parquet for data processing, Delta files are usually larger due to the compression and version history. IMO the versioning features are quite important and it deserves wider adoption outside of the Spark ecosystem.

I have no idea how the compacting and re-partitioning works with non-spark implementation? This feels like the responsibility of some kind of DB or data processing engine, it's probably too much for the Dataset abstraction. WDYT?

[everdark]

By adding this into kedro-datasets, there will be 3 possible ways of handling delta table:
Apache Spark
delta-rs, a non-Spark approach (this PR)
Databricks Unity Catalog

Agree, although DeltaTable is often associated with Spark, but it's actually just a file format and you can read it via Pandas or maybe other libraries later.

I think the current space is still dominated by Parquet for data processing, Delta files are usually larger due to the compression and version history. IMO the versioning features are quite important and it deserves wider adoption outside of the Spark ecosystem.

I have no idea how the compacting and re-partitioning works with non-spark implementation? This feels like the responsibility of some kind of DB or data processing engine, it's probably too much for the Dataset abstraction. WDYT?

It will be mostly based on Apache Arrow. It will make sure we don't need to load all files into memory.
Advanced use case can be one leveraging DuckDB as the in-memory query engine, and with Arrow as the backend.

Some examples (in terms of analytical query) can be found here:
https://delta-io.github.io/delta-rs/python/usage.html#querying-delta-tables

[SajidAlamQB]

Thanks for your contribution @afaqueahmad7117! Your DeltaTableDataSet class looks great.

[afaqueahmad7117]

Thanks for your contribution @afaqueahmad7117! Your DeltaTableDataSet class looks great.

Thank you! @everdark too!

[noklam]

Thank you both! @afaqueahmad7117 @everdark, I will take over from now to fix the conflict and merge.

[noklam]

Moving requirements around as we moved from setup.pu -> pyproject.toml

[astrojuanlu]

Let's get this merged 🚀 thanks @afaqueahmad7117 @everdark and team!