[BUG] Writes from empty partitions should return empty micropartitions with non-null schema

[colin-ho]

If one partition is empty the write will return a list of file paths / partition cols but the data type is NULL. This is problematic because it will cause schema mismatch with other partitions that did have writes.

import daft

df = (
    daft.from_pydict({"foo": [1, 2, 3], "bar": ["a", "b", "c"]})
    .into_partitions(4)
    .write_parquet("z", partition_cols=["bar"])
)
print(df)

daft.exceptions.DaftCoreException: DaftError::SchemaMismatch MicroPartition concat requires all schemas to match, ╭─────────────┬──────╮
│ Column Name ┆ Type │
╞═════════════╪══════╡
│ path        ┆ Utf8 │
├╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌┤
│ bar         ┆ Utf8 │
╰─────────────┴──────╯
 vs ╭─────────────┬──────╮
│ Column Name ┆ Type │
╞═════════════╪══════╡
│ path        ┆ Null │
╰─────────────┴──────╯

[codspeed-hq]

CodSpeed Performance Report

Merging #2952 will not alter performance

_{Comparing colin/fix-writes-with-empty-parts (5b6873a) with main (f1194b5)}

Summary

✅ 17 untouched benchmarks

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 77.99%. Comparing base (28e72b2) to head (5b6873a).
Report is 11 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2952      +/-   ##
==========================================
- Coverage   78.38%   77.99%   -0.39%     
==========================================
  Files         596      598       +2     
  Lines       69693    70742    +1049     
==========================================
+ Hits        54627    55175     +548     
- Misses      15066    15567     +501     

Flag	Coverage Δ
	77.99% <100.00%> (-0.39%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
daft/iceberg/iceberg_write.py	76.47% <100.00%> (+1.03%)	⬆️
daft/table/table_io.py	85.96% <100.00%> (+0.25%)	⬆️

... and 55 files with indirect coverage changes

[kevinzwang]

Looks good! Thanks for fixing this