[FEAT] Read from LanceDB #2195
[FEAT] Read from LanceDB #2195
Conversation
| @@ -165,68 +159,3 @@ def can_absorb_limit(self) -> bool: | |||
|
|
|||
| def can_absorb_select(self) -> bool: | |||
| return True | |||
|
|
|||
|
|
|||
| def _storage_config_to_storage_options(storage_config: StorageConfig, table_uri: str) -> dict[str, str] | None: | |||
There was a problem hiding this comment.
I realized that this stuff is shared between Python libraries that leverage the object_store crate, so I pulled it out to a shared utility that is now used for both DeltaLake and Lance
daft/io/_lance.py
Outdated
| return [ | ||
| Table.from_arrow_record_batches([rb], rb.schema)._table | ||
| for rb in fragment.to_batches(columns=required_columns) | ||
| ] |
There was a problem hiding this comment.
I opted to return a list of Tables, instead of creating one Table for all the record batches to avoid concatenation costs. I think this is fine?
There was a problem hiding this comment.
commented on this below, you should instead return an Iterator[Tables] so we can terminate early but also have the ability to skip the concat.
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2195 +/- ##
=======================================
Coverage ? 85.42%
=======================================
Files ? 71
Lines ? 7514
Branches ? 0
=======================================
Hits ? 6419
Misses ? 1095
Partials ? 0
|
Adds the ability to read a Lance dataset: https://lancedb.github.io/lance/read_and_write.html