2003 - PoC: ibis support #2004

sh-rp · 2024-10-30T12:48:04Z

Description

First draft of ibis support to figure out how we would like to implement this. Please also read the attached ticket for the considered approaches.

Ibis tables should work for all destinations, full ibis connection now is implemented for postgres, filesystem and duckdb in this PoC

Discussion at ibis about using ibis only as a query builder: ibis-project/ibis#10452

netlify · 2024-10-30T12:48:18Z

✅ Deploy Preview for dlt-hub-docs canceled.

Name	Link
🔨 Latest commit	`b3af76b`
🔍 Latest deploy log	https://app.netlify.com/sites/dlt-hub-docs/deploys/672d043053e5100008985929

sh-rp · 2024-10-30T13:56:01Z

dlt/pipeline/pipeline.py

@@ -1729,3 +1734,11 @@ def _dataset(self, dataset_type: TDatasetType = "dbapi") -> SupportsReadableData
            schema=(self.default_schema if self.default_schema_name else None),
            dataset_type=dataset_type,
        )
+
+    def _ibis(self) -> IbisBackend:


I don't think our original idea of exposing an ibis dataset via the same methods as the dbapi one makes sense, the interface of ibis is completely different to ours.

but you can use overloads on Literal dataset_type

@overload def _dataset(self, dataset_type: TDatasetType = "dbapi") -> SupportsReadableDataset: ... @overload def _dataset(self, dataset_type: TDatasetType = "ibis") -> IbisBackend: ...

to return different types. hmmm but maybe you are right. we should implement readable dataset interface on ibis.

so maybe we add ibis() on DBAPI dataset? that will return ibis backend? I do not think adding it on a relation makes sense

I like the overload, I did not know this was possible...

sh-rp · 2024-10-30T13:57:12Z

dlt/destinations/dataset.py

+        raise NotImplementedError()
+
+    ibis = ibis.connect(client.config.credentials.to_native_representation())
+    # NOTE: there seems to be no standardized way to set the current dataset / schema in ibis


I'm not sure where we should put the stuff that converts our credentials into something IBIS understands. client.config.credentials.to_native_representation() could work for many cases, but selecting the default dataset will probably be something destination specific..

Also for the filesystem destination this will work quite differently, because we first need to populate the duckdb database and then attach ibis to it.

part that creates backend should go to dlt/helpers/ibis

part that converts credentials should go to common/libs/ibis
here we can improve a lot. we already pollute credentials in common with concrete converting functions (like fsspec, delta etc.) we should create some universal mechanism to convert them.

rudolfix · 2024-11-01T16:55:08Z

dlt/destinations/impl/filesystem/sql_client.py

@@ -165,13 +165,18 @@ def open_connection(self) -> duckdb.DuckDBPyConnection:
            # set up dataset
            if not self.has_dataset():
                self.create_dataset()
+                print("CREATE")


rudolfix · 2024-11-01T17:07:06Z

dlt/pipeline/pipeline.py

@@ -1729,3 +1734,11 @@ def _dataset(self, dataset_type: TDatasetType = "dbapi") -> SupportsReadableData
            schema=(self.default_schema if self.default_schema_name else None),
            dataset_type=dataset_type,
        )
+
+    def _ibis(self) -> IbisBackend:


but you can use overloads on Literal dataset_type

@overload def _dataset(self, dataset_type: TDatasetType = "dbapi") -> SupportsReadableDataset: ... @overload def _dataset(self, dataset_type: TDatasetType = "ibis") -> IbisBackend: ...

to return different types. hmmm but maybe you are right. we should implement readable dataset interface on ibis.

so maybe we add ibis() on DBAPI dataset? that will return ibis backend? I do not think adding it on a relation makes sense

rudolfix · 2024-11-01T17:10:40Z

dlt/common/destination/reference.py

+        """fetch arrow table of first 'chunk_size' items"""
+        ...
+
+    def ibis(self, chunk_size: int = None) -> Optional[IbisTable]: ...


I'd add it on ReadableDBAPIDataset and return implementation of it in _dataset of pipeline

rudolfix · 2024-11-01T17:11:37Z

dlt/destinations/dataset.py

@@ -42,18 +90,91 @@ def __init__(
        self.iter_arrow = self._wrap_iter("iter_arrow")  # type: ignore
        self.iter_fetch = self._wrap_iter("iter_fetch")  # type: ignore

+    # TODO: where should this go, should cursors support "native" ibis or do we do a conversion somewhere
+    def ibis(self, chunk_size: int = None) -> Optional[IbisTable]:


this does not make sense... we have stuff materialized. way better to just do df() which has data frame interface

I removed those

rudolfix · 2024-11-01T17:12:30Z

dlt/common/destination/reference.py

+        """iterate over arrow tables of 'chunk_size' items"""
+        ...
+
+    def iter_ibis(self, chunk_size: int) -> Generator[IbisTable, None, None]: ...


how are you going to implement that? you'll need to create IbisTable on a chunk of data which is lazy evaluated. otherwise does not make sense

rudolfix · 2024-11-01T17:18:17Z

dlt/destinations/dataset.py

+        raise NotImplementedError()
+
+    ibis = ibis.connect(client.config.credentials.to_native_representation())
+    # NOTE: there seems to be no standardized way to set the current dataset / schema in ibis


part that creates backend should go to dlt/helpers/ibis

part that converts credentials should go to common/libs/ibis
here we can improve a lot. we already pollute credentials in common with concrete converting functions (like fsspec, delta etc.) we should create some universal mechanism to convert them.

rudolfix · 2024-11-01T17:22:33Z

dlt/destinations/dataset.py

+
+    def head(self) -> "ReadableDBAPIRelation":
+        return self.limit(5)
+

 class ReadableDBAPIDataset(SupportsReadableDataset):


let's move ibis() here. then we hand over the connection and return the ibis backend

can we move schema property to the Protocol?

rudolfix · 2024-11-01T17:25:24Z

dlt/destinations/dataset.py

+        raise NotImplementedError()
+
+    # NOTE: there seems to be no standardized way to set the current dataset / schema in ibis
+    ibis.raw_sql(f"SET search_path TO {dataset_name};")


why? you just do:
return ibis[dataset_name]
to select namespace.
https://duckdb.org/docs/guides/python/ibis.html

where does it say that on this page? In any case it does not work and I also looked for a bit before and could not find any standard way to select the current database (as it is called in ibis). The call the whole thing (in the case of duckdb the file) a catalog and what we call dataset is called database there. You can list the database but not set a default one.. At least I have not found a method.

rudolfix · 2024-11-01T19:25:10Z

also here you wrote a helper: https://github.com/dlt-hub/dlt/pull/1491/files

sh-rp self-assigned this Oct 30, 2024

sh-rp linked an issue Oct 30, 2024 that may be closed by this pull request

ibis support for datasets / destinations #2003

Open

sh-rp commented Oct 30, 2024

View reviewed changes

sh-rp force-pushed the feat/2003-ibis-support branch from 2adb193 to 5befef9 Compare October 31, 2024 12:10

sh-rp changed the title ~~2003 - Experiment: ibis support~~ 2003 - PoC: ibis support Oct 31, 2024

rudolfix requested changes Nov 1, 2024

View reviewed changes

sh-rp force-pushed the feat/2003-ibis-support branch 3 times, most recently from 59b0977 to ab62c59 Compare November 7, 2024 15:19

sh-rp added 8 commits November 7, 2024 16:19

add PoC for ibis table support on readabledbapidataset

f4d47f3

add PoC for exposing an ibis backend for a destination

b82aef0

install ibis dependency for tests

a928dc7

add support for filesystem

fdb78dd

remove print statments

4113d52

remove ibis tables from dbapirelation

c09f19b

clean up interfaces

b4e2dfd

move backend creation and skip tests for unsupported backend

500b5ff

sh-rp force-pushed the feat/2003-ibis-support branch from ab62c59 to 500b5ff Compare November 7, 2024 15:19

sh-rp added 2 commits November 7, 2024 19:04

fix dependencies and typing

64de75d

mark import not found, can't be linted on 3.8 and 3.9

b3af76b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

2003 - PoC: ibis support #2004

2003 - PoC: ibis support #2004

sh-rp commented Oct 30, 2024 •

edited

Loading

netlify bot commented Oct 30, 2024 •

edited

Loading

sh-rp Oct 30, 2024

rudolfix Nov 1, 2024

sh-rp Nov 1, 2024

sh-rp Oct 30, 2024

sh-rp Oct 30, 2024

rudolfix Nov 1, 2024

rudolfix Nov 1, 2024

rudolfix Nov 1, 2024

rudolfix Nov 1, 2024

rudolfix Nov 1, 2024

sh-rp Nov 7, 2024

rudolfix Nov 1, 2024

sh-rp Nov 7, 2024

rudolfix Nov 1, 2024

rudolfix Nov 1, 2024

rudolfix Nov 1, 2024

rudolfix Nov 1, 2024

sh-rp Nov 2, 2024

rudolfix commented Nov 1, 2024

2003 - PoC: ibis support #2004

Are you sure you want to change the base?

2003 - PoC: ibis support #2004

Conversation

sh-rp commented Oct 30, 2024 • edited Loading

Description

netlify bot commented Oct 30, 2024 • edited Loading

✅ Deploy Preview for dlt-hub-docs canceled.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rudolfix commented Nov 1, 2024

sh-rp commented Oct 30, 2024 •

edited

Loading

netlify bot commented Oct 30, 2024 •

edited

Loading