Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEAT] Add Unity Catalog support #2377

Merged
merged 6 commits into from
Jun 14, 2024
Merged

[FEAT] Add Unity Catalog support #2377

merged 6 commits into from
Jun 14, 2024

Conversation

jaychia
Copy link
Contributor

@jaychia jaychia commented Jun 14, 2024

  1. Adds a new daft.io.UnityCatalog class
  2. Adds some basic methods on that class, only implements list_schemas, list_tables and load_tables right now
  3. Adds integrations with daft.read_delta_lake to make this work
  4. Ensure that the io_config is correctly propagated by requesting credentials from unity catalog
image

@github-actions github-actions bot added the enhancement New feature or request label Jun 14, 2024
Copy link

codecov bot commented Jun 14, 2024

Codecov Report

Attention: Patch coverage is 29.41176% with 48 lines in your changes missing coverage. Please review.

Please upload report for BASE (main@2290d03). Learn more about missing BASE report.
Report is 3 commits behind head on main.

Current head 3ea4170 differs from pull request most recent head d091f52

Please upload reports for the commit d091f52 to get more accurate results.

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##             main    #2377   +/-   ##
=======================================
  Coverage        ?   63.48%           
=======================================
  Files           ?      909           
  Lines           ?   102088           
  Branches        ?        0           
=======================================
  Hits            ?    64808           
  Misses          ?    37280           
  Partials        ?        0           
Files Coverage Δ
daft/io/_delta_lake.py 72.22% <36.36%> (ø)
daft/io/unity_catalog.py 28.07% <28.07%> (ø)

@jaychia jaychia requested a review from samster25 June 14, 2024 08:52
schema_names.extend([s.full_name for s in response.schemas])

# Exhaust pages
while response.next_page_token is not None and response.next_page_token != "":
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can prob make this a generic util function that creates an iterator from a method / function using this paradigm

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, done

@jaychia jaychia enabled auto-merge (squash) June 14, 2024 21:56
@github-actions github-actions bot added the documentation Improvements or additions to documentation label Jun 14, 2024
@jaychia jaychia merged commit 395ebe8 into main Jun 14, 2024
43 checks passed
@jaychia jaychia deleted the jay/catalog branch June 14, 2024 22:37
@djouallah
Copy link
Contributor

what would be really cool if you can add daft.read_table(table).show(), I don't needs to know if it is delta/iceberg or something else, daft can figure out which table format it is :)

@jaychia
Copy link
Contributor Author

jaychia commented Jun 17, 2024

@djouallah that's a great idea!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants