Build: Bump Apache Parquet 1.14.4 #11502
Conversation
…)" (apache#11462)" This reverts commit 7cc16fa.
Fokko
changed the title
| Row booleanCol = Row.of(36L, 4L, 0L, null, false, true); | ||
| Row decimalCol = Row.of(91L, 4L, 1L, null, new BigDecimal("1.00"), new BigDecimal("2.00")); | ||
| Row doubleCol = Row.of(91L, 4L, 0L, 1L, 1.0D, 2.0D); |
There was a problem hiding this comment.
[optional] should we refactor this to pick file_size from the Datafiles themselve like we did we did in JDK 17 upgrade PR #7391 (comment)
Never the less looks like size in bytes is increasing in this version is it because they are more accurate now ?
There was a problem hiding this comment.
Hey @singhpk234, that's an excellent suggestion. I've copied your approach here as well. Parquet now also tracks how large the data is in memory after compression (this is handy for strings where you don't know that upfront) so you can allocate buffers directly to the right size.
There was a problem hiding this comment.
how large the data is in memory after compression (this is handy for strings where you don't know that upfront) so you can allocate buffers directly to the right size.
This is precisely what we needed in Redshift as well, our CBO was falling behind with variable length data types, will give them HeadsUp ! Thankyou @Fokko
build.gradle
Outdated
| @@ -119,6 +119,9 @@ allprojects { | |||
| repositories { | |||
| mavenCentral() | |||
| mavenLocal() | |||
| maven { | |||
| url = uri("https://repository.apache.org/content/repositories/orgapacheparquet-1065") | |||
There was a problem hiding this comment.
Just a note: this is temporary during the Parquet release vote (just to not forget to remove this once Parquet release is out 😄 )
| @@ -217,27 +217,27 @@ public void testPrimitiveColumns() throws Exception { | |||
|
|
|||
| Row binaryCol = | |||
| Row.of( | |||
| 52L, | |||
| 55L, | |||
There was a problem hiding this comment.
Why the size is growing here (I mean in the test) ?
Should we have two tests ?
There was a problem hiding this comment.
See #11502 (comment) for the reason. What's the suggestion for the second test?
| Row fixedCol = | ||
| Row.of( | ||
| 44L, | ||
| 47L, |
There was a problem hiding this comment.
Same question here about the size.
Fokko
changed the title
No description provided.