-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Java] Added creating MapWriter in ComplexWriter. #36375
Comments
xxlaykxx
changed the title
Added creating MapWriter in ComplexWriter.
[Java] Added creating MapWriter in ComplexWriter.
Jun 29, 2023
lidavidm
pushed a commit
that referenced
this issue
Jul 5, 2023
Added new method rootAsMap() to ComplexWriter and implement it in ComplexWriterImpl for supporting map type. Previously in dremio side: When i trying to return map like output ComplexWrite with this code: org.apache.arrow.vector.complex.writer.BaseWriter.MapWriter mapWriter = out.rootAsList().map(false); mapWriter.startMap(); for (java.util.Map.Entry<java.lang.Integer, java.lang.Integer> element : map.entrySet()) { mapWriter.startEntry(); mapWriter.key().integer().writeInt((Integer) element.getKey()); mapWriter.value().integer().writeInt((Integer) element.getValue()); mapWriter.endEntry(); } mapWriter.endMap(); It use UnionMapWriter and generate schema like: EXPR$0: Map(false)<$data$: Union(Sparse, [1, 39])<struct: Struct<key: Int(32, true) not null, value: Int(32, true) not null> not null, map: Map(false)<entries: Struct<key: Int(32, true) not null, value: Int(32, true)> not null>>> But in OutputDerivation impl class where i should create output Complete type List<Field> children = Arrays.asList( CompleteType.INT.toField("key", false), CompleteType.INT.toField("value", false)); return new CompleteType(CompleteType.MAP.getType(), CompleteType.struct(children).toField(MapVector.DATA_VECTOR_NAME, false)); (This is only one valid case, because MapVector.initializeChildrenFromFields()) return EXPR$0::map<key::int32, value::int32> I found a place where it start using union - PromotableWriter.promoteToUnion. And in the end i have SCHEMA_CHANGE ERROR: Schema changed during projection. Schema was schema(EXPR$0::map<key::int32, value::int32>) but then changed to schema(EXPR$0::map<struct::struct<key::int32, value::int32>, map::map<key::int32, value::int32>>) * Closes: #36375 Authored-by: Ivan Chesnov <[email protected]> Signed-off-by: David Li <[email protected]>
xxlaykxx
added a commit
to dremio/arrow
that referenced
this issue
Jul 7, 2023
…ache#36351) (#32) Added new method rootAsMap() to ComplexWriter and implement it in ComplexWriterImpl for supporting map type. Previously in dremio side: When i trying to return map like output ComplexWrite with this code: org.apache.arrow.vector.complex.writer.BaseWriter.MapWriter mapWriter = out.rootAsList().map(false); mapWriter.startMap(); for (java.util.Map.Entry<java.lang.Integer, java.lang.Integer> element : map.entrySet()) { mapWriter.startEntry(); mapWriter.key().integer().writeInt((Integer) element.getKey()); mapWriter.value().integer().writeInt((Integer) element.getValue()); mapWriter.endEntry(); } mapWriter.endMap(); It use UnionMapWriter and generate schema like: EXPR$0: Map(false)<$data$: Union(Sparse, [1, 39])<struct: Struct<key: Int(32, true) not null, value: Int(32, true) not null> not null, map: Map(false)<entries: Struct<key: Int(32, true) not null, value: Int(32, true)> not null>>> But in OutputDerivation impl class where i should create output Complete type List<Field> children = Arrays.asList( CompleteType.INT.toField("key", false), CompleteType.INT.toField("value", false)); return new CompleteType(CompleteType.MAP.getType(), CompleteType.struct(children).toField(MapVector.DATA_VECTOR_NAME, false)); (This is only one valid case, because MapVector.initializeChildrenFromFields()) return EXPR$0::map<key::int32, value::int32> I found a place where it start using union - PromotableWriter.promoteToUnion. And in the end i have SCHEMA_CHANGE ERROR: Schema changed during projection. Schema was schema(EXPR$0::map<key::int32, value::int32>) but then changed to schema(EXPR$0::map<struct::struct<key::int32, value::int32>, map::map<key::int32, value::int32>>) * Closes: apache#36375 Authored-by: Ivan Chesnov <[email protected]> Signed-off-by: David Li <[email protected]>
lriggs
pushed a commit
to dremio/arrow
that referenced
this issue
Jul 13, 2023
…ache#36351) (#32) Added new method rootAsMap() to ComplexWriter and implement it in ComplexWriterImpl for supporting map type. Previously in dremio side: When i trying to return map like output ComplexWrite with this code: org.apache.arrow.vector.complex.writer.BaseWriter.MapWriter mapWriter = out.rootAsList().map(false); mapWriter.startMap(); for (java.util.Map.Entry<java.lang.Integer, java.lang.Integer> element : map.entrySet()) { mapWriter.startEntry(); mapWriter.key().integer().writeInt((Integer) element.getKey()); mapWriter.value().integer().writeInt((Integer) element.getValue()); mapWriter.endEntry(); } mapWriter.endMap(); It use UnionMapWriter and generate schema like: EXPR$0: Map(false)<$data$: Union(Sparse, [1, 39])<struct: Struct<key: Int(32, true) not null, value: Int(32, true) not null> not null, map: Map(false)<entries: Struct<key: Int(32, true) not null, value: Int(32, true)> not null>>> But in OutputDerivation impl class where i should create output Complete type List<Field> children = Arrays.asList( CompleteType.INT.toField("key", false), CompleteType.INT.toField("value", false)); return new CompleteType(CompleteType.MAP.getType(), CompleteType.struct(children).toField(MapVector.DATA_VECTOR_NAME, false)); (This is only one valid case, because MapVector.initializeChildrenFromFields()) return EXPR$0::map<key::int32, value::int32> I found a place where it start using union - PromotableWriter.promoteToUnion. And in the end i have SCHEMA_CHANGE ERROR: Schema changed during projection. Schema was schema(EXPR$0::map<key::int32, value::int32>) but then changed to schema(EXPR$0::map<struct::struct<key::int32, value::int32>, map::map<key::int32, value::int32>>) * Closes: apache#36375 Authored-by: Ivan Chesnov <[email protected]> Signed-off-by: David Li <[email protected]>
lriggs
added a commit
to dremio/arrow
that referenced
this issue
Jul 25, 2023
…ter. (apache#36351) (#32)" This reverts commit 0e7d9b5. This branch is inteded to work with Dremio 24.2 and it does not include the necessary Dremio changes for BaseWriter.
xxlaykxx
added a commit
to dremio/arrow
that referenced
this issue
Jul 30, 2023
* apacheGH-36375: [Java] Added creating MapWriter in ComplexWriter. (apache#36351) (#32) Added new method rootAsMap() to ComplexWriter and implement it in ComplexWriterImpl for supporting map type. Previously in dremio side: When i trying to return map like output ComplexWrite with this code: org.apache.arrow.vector.complex.writer.BaseWriter.MapWriter mapWriter = out.rootAsList().map(false); mapWriter.startMap(); for (java.util.Map.Entry<java.lang.Integer, java.lang.Integer> element : map.entrySet()) { mapWriter.startEntry(); mapWriter.key().integer().writeInt((Integer) element.getKey()); mapWriter.value().integer().writeInt((Integer) element.getValue()); mapWriter.endEntry(); } mapWriter.endMap(); It use UnionMapWriter and generate schema like: EXPR$0: Map(false)<$data$: Union(Sparse, [1, 39])<struct: Struct<key: Int(32, true) not null, value: Int(32, true) not null> not null, map: Map(false)<entries: Struct<key: Int(32, true) not null, value: Int(32, true)> not null>>> But in OutputDerivation impl class where i should create output Complete type List<Field> children = Arrays.asList( CompleteType.INT.toField("key", false), CompleteType.INT.toField("value", false)); return new CompleteType(CompleteType.MAP.getType(), CompleteType.struct(children).toField(MapVector.DATA_VECTOR_NAME, false)); (This is only one valid case, because MapVector.initializeChildrenFromFields()) return EXPR$0::map<key::int32, value::int32> I found a place where it start using union - PromotableWriter.promoteToUnion. And in the end i have SCHEMA_CHANGE ERROR: Schema changed during projection. Schema was schema(EXPR$0::map<key::int32, value::int32>) but then changed to schema(EXPR$0::map<struct::struct<key::int32, value::int32>, map::map<key::int32, value::int32>>) * Closes: apache#36375 Authored-by: Ivan Chesnov <[email protected]> Signed-off-by: David Li <[email protected]> * DX-67936 Upgrade to Netty 4.1.96 for CVE-2023-34462 io.netty:netty-handler 4.1.93.Final (#36) * Update README_DREMIO for new commit. --------- Signed-off-by: David Li <[email protected]> Co-authored-by: lriggs <[email protected]> Co-authored-by: Logan Riggs <[email protected]>
xxlaykxx
added a commit
to dremio/arrow
that referenced
this issue
Jul 30, 2023
…ache#36351) (#32) (#39) Added new method rootAsMap() to ComplexWriter and implement it in ComplexWriterImpl for supporting map type. Previously in dremio side: When i trying to return map like output ComplexWrite with this code: org.apache.arrow.vector.complex.writer.BaseWriter.MapWriter mapWriter = out.rootAsList().map(false); mapWriter.startMap(); for (java.util.Map.Entry<java.lang.Integer, java.lang.Integer> element : map.entrySet()) { mapWriter.startEntry(); mapWriter.key().integer().writeInt((Integer) element.getKey()); mapWriter.value().integer().writeInt((Integer) element.getValue()); mapWriter.endEntry(); } mapWriter.endMap(); It use UnionMapWriter and generate schema like: EXPR$0: Map(false)<$data$: Union(Sparse, [1, 39])<struct: Struct<key: Int(32, true) not null, value: Int(32, true) not null> not null, map: Map(false)<entries: Struct<key: Int(32, true) not null, value: Int(32, true)> not null>>> But in OutputDerivation impl class where i should create output Complete type List<Field> children = Arrays.asList( CompleteType.INT.toField("key", false), CompleteType.INT.toField("value", false)); return new CompleteType(CompleteType.MAP.getType(), CompleteType.struct(children).toField(MapVector.DATA_VECTOR_NAME, false)); (This is only one valid case, because MapVector.initializeChildrenFromFields()) return EXPR$0::map<key::int32, value::int32> I found a place where it start using union - PromotableWriter.promoteToUnion. And in the end i have SCHEMA_CHANGE ERROR: Schema changed during projection. Schema was schema(EXPR$0::map<key::int32, value::int32>) but then changed to schema(EXPR$0::map<struct::struct<key::int32, value::int32>, map::map<key::int32, value::int32>>) * Closes: apache#36375 Authored-by: Ivan Chesnov <[email protected]> Signed-off-by: David Li <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the enhancement requested
Add new method rootAsMap() to ComplexWriter and implement it in ComplexWriterImpl for supporting Map type. Because currently it's possible to create MapWriter but like a part of list or struct.
Previously in dremio side:
When i trying to return map like @output ComplexWrite
with this code:
It use UnionMapWriter and generate schema like:
EXPR$0: Map(false)<$data$: Union(Sparse, [1, 39])<struct: Struct<key: Int(32, true) not null, value: Int(32, true) not null> not null, map: Map(false)<entries: Struct<key: Int(32, true) not null, value: Int(32, true)> not null>>>
But in OutputDerivation impl class where i should create output Complete type
(This is only one valid case, because MapVector.initializeChildrenFromFields())
return
EXPR$0::map<key::int32, value::int32> I found a place where it start using union - PromotableWriter.promoteToUnion.
And in the end i have
SCHEMA_CHANGE ERROR: Schema changed during projection. Schema was
schema(EXPR$0::map<key::int32, value::int32>)
but then changed to
schema(EXPR$0::map<struct::struct<key::int32, value::int32>, map::map<key::int32, value::int32>>)
Component(s)
Java
The text was updated successfully, but these errors were encountered: