-
Notifications
You must be signed in to change notification settings - Fork 159
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FEAT] Add Unity Catalog support #2377
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2377 +/- ##
=======================================
Coverage ? 63.48%
=======================================
Files ? 909
Lines ? 102088
Branches ? 0
=======================================
Hits ? 64808
Misses ? 37280
Partials ? 0
|
daft/io/unity_catalog.py
Outdated
schema_names.extend([s.full_name for s in response.schemas]) | ||
|
||
# Exhaust pages | ||
while response.next_page_token is not None and response.next_page_token != "": |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we can prob make this a generic util function that creates an iterator from a method / function using this paradigm
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice, done
what would be really cool if you can add daft.read_table(table).show(), I don't needs to know if it is delta/iceberg or something else, daft can figure out which table format it is :) |
@djouallah that's a great idea! |
daft.io.UnityCatalog
classlist_schemas
,list_tables
andload_tables
right nowdaft.read_delta_lake
to make this workio_config
is correctly propagated by requesting credentials from unity catalog