Prevent multiple files per partition in TestMetadataQueryOptimization

[sopel39]

optimize_metadata_queries doesn't work when files are written by different writers

Fixes #15507

[sopel39]

Scaling shouldn't happen here as CTAS is very small

                "CREATE TABLE %s (a, b, c) WITH (PARTITIONING = ARRAY['b', 'c']) AS VALUES (5, 6, 7), (8, 9, 10)",

locally this test passes without scaling (and scaling should never trigger because exchange buffer is much bigger than 6 rows)

[sopel39]

Managed to repro with 2 writers, it seems that optimize_metadata_queries doesn't work when files are written by different writers

[sopel39]

Adjuster PR

[findepi]

Thanks for fixing the failing test!

optimize_metadata_queries doesn't work when files are written by different writers

this sounds like a product bug, while fix looks like test-only change.
what am i missing?

[sopel39]

this sounds like a product bug, while fix looks like test-only change.
what am i missing?

I don't know Iceberg internals. For me it's odd that this optimization stops working when partitions are written by different writers. I'm leaving it to you @findepi to triage it. My goal here was to restore previous behavior in test and unblock master.

[findepi]

My goal here was to restore previous behavior in test and unblock master.

yes, and that's appreciated!

For me it's odd that this optimization stops working when partitions are written by different writers

sounds odd, indeed
which issue tracks this?

[sopel39]

which issue tracks this?

none. Again, I'm not sure if its a bug, therefore I defer to more Iceberg experienced person for triaging.