[BUG] Fix intersection checking when unioning schemas

[desmondcheongzx]

In the definition of Schema::union, the error message suggests that we intended to throw errors when performing a union on two schemas with overlapping keys. However, the original implementation took the set difference of keys between one side of the union and itself, which would never throw an error.

This bug was not noticed because the python tests went through the python code path which would check for the intersection correctly. But if one uses the Rust API directly, then this property is not upheld.

We fix this bug by instead checking that the two sides of the union have distinct keys.

[codspeed-hq]

CodSpeed Performance Report

Merging #3039 will not alter performance

_{Comparing desmondcheongzx:fix-schema-union (f3ba7b1) with main (a3453d1)}

Summary

✅ 17 untouched benchmarks

[kevinzwang]

Looks like it's failing some parquet integration tests

[kevinzwang]

When do we have two tables that we are unioning that have common columns? Just want to make sure that this non-distinct union is the correct behavior.

[desmondcheongzx]

This really only gets used in the MicroPartition API for reading multiple parquet files. E.g. daft.table.MicroPartition.read_parquet_bulk(["file1.parquet", "file2.parquet"])

Here both files can have the same columns.

The other MicroPartition APIs for read_parquet, read_csv, and read_json are non-concerns because they only ever take in one uri. But the read_{csv, json, parquet}_into_micropartition functions they call under the hood take in a slice of uris and can run into the same problem that read_parquet_bulk currently does. As of today there are no other users besides read_parquet_bulk that read more than one uri.

FWIW I believe the original authors (@jaychia and @clarkzinzow) intended to use the semantics of a non-distinct union. But I'm not 100% sure why we would bother with the cases where the schemas were mismatched---I imagine this would quickly blow up elsewhere.