Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CHORE] Enable lancedb reads for native executor #2925

Merged
merged 17 commits into from
Oct 17, 2024

Conversation

colin-ho
Copy link
Contributor

@colin-ho colin-ho commented Sep 25, 2024

Enables Lance reads for swordfish.

The existing lance read mechanism reads from python iterators. This PR abstracts out that logic to a separate function.

Both the python and streaming executors will call this function, the python executor will do a .collect on the iterator, while the native runner will wrap the iterator in a stream.

@github-actions github-actions bot added the chore label Sep 25, 2024
Copy link

codspeed-hq bot commented Sep 25, 2024

CodSpeed Performance Report

Merging #2925 will not alter performance

Comparing colin/swordfish-lance-reads (0149883) with main (d243cee)

Summary

✅ 17 untouched benchmarks

Copy link

codecov bot commented Sep 25, 2024

Codecov Report

Attention: Patch coverage is 95.06173% with 4 lines in your changes missing coverage. Please review.

Project coverage is 78.32%. Comparing base (d243cee) to head (0149883).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
src/daft-micropartition/src/python.rs 94.73% 4 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##             main    #2925   +/-   ##
=======================================
  Coverage   78.32%   78.32%           
=======================================
  Files         611      611           
  Lines       72421    72427    +6     
=======================================
+ Hits        56725    56730    +5     
- Misses      15696    15697    +1     
Files with missing lines Coverage Δ
src/daft-local-execution/src/sources/scan_task.rs 73.02% <100.00%> (+1.07%) ⬆️
src/daft-micropartition/src/micropartition.rs 90.50% <100.00%> (-0.33%) ⬇️
src/daft-micropartition/src/python.rs 78.23% <94.73%> (+1.55%) ⬆️

... and 3 files with indirect coverage changes

@colin-ho colin-ho marked this pull request as ready for review October 8, 2024 23:33
@colin-ho colin-ho enabled auto-merge (squash) October 17, 2024 17:27
@colin-ho colin-ho merged commit 69fef20 into main Oct 17, 2024
40 checks passed
@colin-ho colin-ho deleted the colin/swordfish-lance-reads branch October 17, 2024 17:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants