feat: support type coercion in Parquet Reader

[e1ijah1]

Which issue does this PR close?

Closes #6427 .

Rationale for this change

What changes are included in this PR?

Support type coercion in Parquet Reader

Are these changes tested?

Are there any user-facing changes?

[alamb]

Thanks @e1ijah1 ! This is looking very cool

I think to really complete this feature we should have an "end to end test" -- like actually creating parquet files with two different schemas and showing how they can be read as a single table using this feature

Perhaps we could add a test to https://github.com/apache/arrow-datafusion/blob/main/datafusion/core/tests/parquet, following the model of

https://github.com/apache/arrow-datafusion/blob/main/datafusion/core/tests/parquet/custom_reader.rs

@tustvold since you filed #6427 do you have some idea about how this feature would be used?

[alamb]

Marking as draft as we are waiting on feedback

[tustvold]

do you have some idea about how this feature would be used

The use-case is where you have one or more parquet files, with different schema. You can provide a file_schema to FileScanConfig (or to ListingTableConfig) and have underlying data coerced to that schema on read. An example might be if a column has changed from a StringArray to DictionaryArray or vice-versa

[e1ijah1]

Thank you for your valuable feedback, @alamb and @tustvold ❤️. I've added an end-to-end test to ensure the type coercion functionality in the Parquet reader works as expected. Please let me know if there are any other improvements you'd like to see.

[tustvold]

Thank you this looks good to me

[tustvold]

Getting this in to avoid conflicts with #6374

[tustvold]

I filed a follow on PR #6563 that avoids needing to recompute the mapping for each batch