Support complex types in Arrow/Parquet/ORC

[Avogar]

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

Changelog category (leave one):

• New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Support structs and maps in Arrow/Parquet/ORC and dictionaries in Arrow input/output formats. Present new setting output_format_arrow_low_cardinality_as_dictionary.

Detailed description / Documentation draft:
Support Struct and Map types in input/output column-oriented formats Arrow/Parquet/ORC. Now you can input/output ClickHouse Tuples and Maps (experimental type) in these formats. Nested complex types are also supported. If setting output_format_arrow_low_cardinality_as_dictionary is true, LowCardinality columns will be converted into dictionary Arrow column, if false, LowCardinality columns will be converted to full column before output. By default this setting is false.
Closes #17240 and #21866

[Avogar]

@Mergifyio update

[mergify]

Command update: success

Branch has been successfully updated

[Avogar]

I am also going to add Map type support in this PR.

[Avogar]

@Mergifyio update

[mergify]

Command update: success

Branch has been successfully updated

[buyology]

Thanks so much for working on this.

I tried this PR as I thought it could help us to load some of our Parquet files into ClickHouse.

The table

:) CREATE TABLE t ( v Nested(a String, b String) ) ENGINE=Memory()

An array of structs

» parquet-tools schema f1
message schema {
  required group v (LIST) {
    repeated group list {
      required group v {
        optional binary a (STRING);
        optional binary b (STRING);
      }
    }
  }
}

» ch --query="INSERT INTO t FORMAT Parquet" < f1
Code: 8. DB::Exception: Column "v.a" is not presented in input data.: data for INSERT was parsed from stdin

A struct of arrays

» parquet-tools schema f2
message schema {
  optional group v {
    required group a (LIST) {
      repeated group list {
        required binary a (STRING);
      }
    }
    required group b (LIST) {
      repeated group list {
        required binary b (STRING);
      }
    }
  }
}

» ch --query="INSERT INTO t FORMAT Parquet" < f2
Code: 8. DB::Exception: Column "v.a" is not presented in input data.: data for INSERT was parsed from stdin

This could be worked around by transforming the data using input-function (going through Tuples), but it would be amazing to just be able to load both of these files directly into the destination table.

[Avogar]

@Mergifyio update

[mergify]

Command update: success

Branch has been successfully updated

[Avogar]

@buyology thank you for you comment, I will try to support inserting into Nested by struct of arrays or arrays of struct in the next PR.

[Avogar]

AST fuzzer failure: #25293

[Avogar]

@Mergifyio update

[mergify]

Command update: success

Branch has been successfully updated

[sevirov]

Internal documentation ticket: DOCSUP-10557.

[alexey-milovidov]

Continued here: #36832