[BUG] test_delta_part_write_round_trip_unmanaged and test_delta_multi_part_write_round_trip_unmanaged fail with DATAGEN_SEED=1700105176

[ttnghia]

They are new failed tests on databricks, with DATAGEN_SEED=1700105176:

FAILED ../../src/main/python/delta_lake_write_test.py::test_delta_part_write_round_trip_unmanaged[Float] - AssertionError: Different line counts in 00000000000000000000.json
FAILED ../../src/main/python/delta_lake_write_test.py::test_delta_part_write_round_trip_unmanaged[Double] - AssertionError: Different line counts in 00000000000000000000.json
FAILED ../../src/main/python/delta_lake_write_test.py::1mtest_delta_multi_part_write_round_trip_unmanaged[Float] - AssertionError: Different line counts in 00000000000000000000.json
FAILED ../../src/main/python/delta_lake_write_test.py::1mtest_delta_multi_part_write_round_trip_unmanaged[Double] - AssertionError: Different line counts in 00000000000000000000.json

[jlowe]

The tests are failing in the delta log check because, for some weird reason, the CPU decides to create lots of very tiny files in the partitions and the GPU only creates one file per partition. In the CPU case, each file contains only between 1-3 records which is crazy and likely a bug. It's pretty weird to use floating point types as partition keys, so maybe not surprising to find odd behaviors here.

[jlowe]

These tests failed on Apache Spark 3.3.3 in a recently nightly build, specifically:

test_delta_part_write_round_trip_unmanaged[Float][DATAGEN_SEED=1700671951]
test_delta_multi_part_write_round_trip_unmanaged[Float][DATAGEN_SEED=1700671951]

[jlowe]

The failure on Apache Spark 3.3.3 involves NaN as a partition value. It appears this triggers Spark CPU to create lots of tiny files, probably because when it checks to see if it's in the same partition when it moves to the next record, NaN != NaN so it thinks it is switching partitions and thus needs to create a new file. Since we're writing batches of data rather than individual rows, we end up creating orders of magnitude fewer files.