Field Metadata Lost on COUNT DISTINCT queries resulting in Internal Error: Physical input schema should be the same as the one converted from logical input schema

[alamb]

Describe the bug

During upgrade of DataFusion in InfluxDB @itsjunetime found that somewhere Field level metadata (aka Field::with_metadata is being lost during DataFusion logical planning

This manifests itself as an error during the physical planning

Internal Error: Physical input schema should be the same as the one converted from logical input schema

@itsjunetime found a workaround in #12631 (which is to ignore the Field metadata). This ticket tracks fixing it for real.

To Reproduce

(we are working on a reproducer)

Expected behavior

No error during physical planning

Additional context

Copying @itsjunetime's description here: #12560 (comment)

I'm running into this behavior after #11989, specifically seeing schema mismatches where the only thing that is different is that a field's metadata disappears at some point (so the schemas are the same except for a field's metadata). E.g.:

&physical_input_schema = Schema {
    fields: [
        Field {
            name: "alias1",
            data_type: Utf8,
            nullable: true,
            dict_id: 0,
            dict_is_ordered: false,
            metadata: {},
        },
    ],
    metadata: {},
}
&physical_input_schema_from_logical = Schema {
    fields: [
        Field {
            name: "alias1",
            data_type: Utf8,
            nullable: true,
            dict_id: 0,
            dict_is_ordered: false,
            metadata: {
                "some_key": "some_value"
            },
        },
    ],
    metadata: {},
}

I've yet to figure out exactly where the metadata is being dropped and I haven't figured out a reproducer either. I suggested comparing only the fields' non-metadata fields here, but @jayzhan211 pointed out that that's more of a workaround than an actual fix, as it's still a problem if the metadata is disappearing.

The issue that I'm running into, though, seems to be somewhat different than the issue that others (like @phillipleblanc) are running into, where some fields completely disappear from the schema (see here). I don't think these are the same issue, exactly (since they manifest differently), but they may have the same root cause/solution, so I think it's fair to keep them all under this issue unless needed otherwise.

I'll work on getting a fix or reproducer today

[alamb]

Here is my suggestion of how to try and reproduce the issue we see:

1. Add a new sqllogictest like metadata.rs or something: https://github.com/apache/datafusion/tree/main/datafusion/sqllogictest
2. Add custom / rust setup code here that creates a table that has schemas with metadata (both metadata on the schema and on the fields):

   datafusion/datafusion/sqllogictest/src/test_context.rs

   Lines 104 to 109 in 792f07d

   	"joins.slt" => {
   	info!("Registering partition table tables");
   	let example_udf = create_example_udf();
   	test_ctx.ctx.register_udf(example_udf);
   	register_partition_table(&mut test_ctx).await;
   	}

3. Add various queries in metadata.slt with the appropriate plan nodes (GROUP BY, ORDER BY, etc) and see if you can provoke the same error.

[alamb]

I am happy to help with the queries if someone can help set up the infrastructure

[wiedld]

take

[alamb]

Labeling this as a regression as it used to work and now does nt

[itsjunetime]

I think this specific issue is fixed, but after #12691, I'm now seeing issues with added field metadata (as opposed to missing field metadata) at the same comparison point. I'll open another issue for it.