[CHORE] Remove non-MicroPartition and non-ScanOperator paths

[clarkzinzow]

This PR removes the legacy non-MicroPartition and non-ScanOperator code paths, refactoring where required. As a drive-by, this PR also refactors the query optimization tests to use structural equality assertions instead of repr-based ones, and adds additional test coverage around scan pushdowns.

Closes #1863
Closes #1939
Closes #1945

[codecov]

Codecov Report

Attention: Patch coverage is 92.59259% with 2 lines in your changes are missing coverage. Please review.

Project coverage is 83.93%. Comparing base (99db777) to head (3ab8ff3).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1946      +/-   ##
==========================================
- Coverage   85.63%   83.93%   -1.71%     
==========================================
  Files          55       55              
  Lines        6250     6111     -139     
==========================================
- Hits         5352     5129     -223     
- Misses        898      982      +84     

Files	Coverage Δ
daft/execution/execution_step.py	93.23% <100.00%> (-0.44%)	⬇️
daft/execution/physical_plan.py	95.17% <100.00%> (+0.12%)	⬆️
daft/execution/rust_physical_plan_shim.py	93.10% <ø> (-0.72%)	⬇️
daft/io/_csv.py	95.23% <100.00%> (ø)
daft/io/_json.py	100.00% <100.00%> (ø)
daft/io/_parquet.py	100.00% <100.00%> (ø)
daft/logical/builder.py	89.83% <100.00%> (-0.34%)	⬇️
daft/logical/schema.py	91.96% <ø> (+0.58%)	⬆️
daft/plan_scheduler/physical_plan_scheduler.py	75.00% <100.00%> (ø)
daft/runners/pyrunner.py	96.62% <100.00%> (+0.45%)	⬆️
... and 5 more

... and 2 files with indirect coverage changes

[samster25]

@clarkzinzow btw: it still looks like github actions is still running the matrix over MICROPARTITIONS

[clarkzinzow]

@samster25 Hmm I thought that I removed that from the actions config, let me double-check.

[jaychia]

Looks great to me!!! I feel like a new man...

Amazing work going through all our optimizer tests 😰 I think the dummy_scan_operator is fine, and makes the tests nice and explicit

[jaychia]

Goodbye and thank you for your service 🫡

[jaychia]

We'll need github repo admin permissions to change our "Required" CI steps before we can actually merge!

[clarkzinzow]

We'll need github repo admin permissions to change our "Required" CI steps before we can actually merge!

@jaychia Not sure if I understand how the required workflows work, but I just rebased this PR on latest main, so I think we can have someone with admin repo privileges force-merge the PR since all pre-merge checks are green, without waiting for the post-merge checks to run. I assume that our required workflows are reusing our normal pre-merge workflows as a "reusable" workflow, so once the reusable workflows are updated in main via merging this PR, main commits and PR commits should be post-merge tested using the updated workflows as expected.

Am I understanding our required workflows setup correctly? I don't think I have permissions to see it, let alone modify it, unfortunately. ☹️

[jaychia]

Oh! I was just suggesting that when we’re ready to merge I can go into the panel and switch everything out so we get a green merge No action items on your end, just lmk when we're ready 😁

[clarkzinzow]

@jaychia should be good to merge now!

[jaychia]

Edited main branch protection rules and merged PR