You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We might use the old DB to generate a file and read it using the new DB. If this is a breaking change, we should find some places to cast the array to the target type.
Note that we should also modify the helper to cast the binary array correctly.
What type of enhancement is this?
Tech debt reduction
What does the enhancement do?
Due to historical reasons, we use
LargeBinaryArray
for the binary type butStringArray
for the string type.greptimedb/src/datatypes/src/arrow_array.rs
Lines 15 to 18 in b1ef327
As storing and processing are inefficient, I suggest that we could change the default binary array type to
BinaryArray
.Implementation challenges
Since the
BinaryArray
is a type alias, we could change it toLargeBinaryArray
and fix the remaining compiler errors.We need to change the data type mapping to the arrow type.
greptimedb/src/datatypes/src/types/binary_type.rs
Lines 49 to 51 in b1ef327
We use our data type and schema while reading the parquet file so we should be able to read old parquet files with binary type columns.
greptimedb/src/mito2/src/sst/parquet/reader.rs
Lines 204 to 208 in b1ef327
We might use the old DB to generate a file and read it using the new DB. If this is a breaking change, we should find some places to cast the array to the target type.
Note that we should also modify the helper to cast the binary array correctly.
greptimedb/src/datatypes/src/vectors/helper.rs
Lines 261 to 266 in b1ef327
I also found a bug while casting the
LargeUtf8
type. We are converting a string array into a binary vector......greptimedb/src/datatypes/src/vectors/helper.rs
Lines 278 to 282 in b1ef327
The text was updated successfully, but these errors were encountered: