Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[PERF] Make merging of ScanTasks be more conservative when provided with a LIMIT #2758

Merged
merged 3 commits into from
Aug 31, 2024

Conversation

jaychia
Copy link
Contributor

@jaychia jaychia commented Aug 28, 2024

No description provided.

@jaychia jaychia changed the title Jay/merge scan task limit pushdowns [PERF] Make merging of ScanTasks be more conservative when provided with a LIMIT Aug 28, 2024
@jaychia jaychia requested a review from samster25 August 28, 2024 23:54
src/daft-scan/src/scan_task_iters.rs Outdated Show resolved Hide resolved
Base automatically changed from jay/fix-memory-estimations-limits-scantask to main August 30, 2024 21:54
@jaychia jaychia force-pushed the jay/merge-scan-task-limit-pushdowns branch from 8242fea to ef5af18 Compare August 30, 2024 22:57
Copy link

codspeed-hq bot commented Aug 30, 2024

CodSpeed Performance Report

Merging #2758 will improve performances by ×8

Comparing jay/merge-scan-task-limit-pushdowns (ef5af18) with main (7e134f2)

Summary

⚡ 2 improvements
✅ 14 untouched benchmarks

Benchmarks breakdown

Benchmark main jay/merge-scan-task-limit-pushdowns Change
test_count[1 Small File] 25.8 ms 23.1 ms +11.36%
test_show[100 Small Files] 393.5 ms 49.3 ms ×8

Copy link

codecov bot commented Aug 30, 2024

Codecov Report

Attention: Patch coverage is 97.56098% with 1 line in your changes missing coverage. Please review.

Please upload report for BASE (main@7e134f2). Learn more about missing BASE report.
Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
src/daft-scan/src/scan_task_iters.rs 97.43% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##             main    #2758   +/-   ##
=======================================
  Coverage        ?   63.23%           
=======================================
  Files           ?      996           
  Lines           ?   113444           
  Branches        ?        0           
=======================================
  Hits            ?    71740           
  Misses          ?    41704           
  Partials        ?        0           
Files with missing lines Coverage Δ
src/daft-plan/src/physical_planner/translate.rs 91.49% <100.00%> (ø)
src/daft-scan/src/scan_task_iters.rs 96.93% <97.43%> (ø)

@jaychia jaychia merged commit 3a7a0b4 into main Aug 31, 2024
37 checks passed
@jaychia jaychia deleted the jay/merge-scan-task-limit-pushdowns branch August 31, 2024 01:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants