You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Latest arrow-memory-netty jar fails to work with latest netty-handler.jar with the following exception.
java.lang.NoSuchMethodError: io/netty/buffer/PooledByteBufAllocatorL$InnerAllocator.threadCache()Lio/netty/buffer/PoolThreadCache; (loaded from file:/opt/ibm/spark/jars/arrow-memory-netty-7.0.0.jar by jdk.internal.loader.ClassLoaders$AppClassLoader@5f26aa33) called from class io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator (loaded from file:/opt/ibm/spark/jars/arrow-memory-netty-7.0.0.jar by jdk.internal.loader.ClassLoaders$AppClassLoader@5f26aa33).
at io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.newDirectBufferL(PooledByteBufAllocatorL.java:164)
at io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.directBuffer(PooledByteBufAllocatorL.java:214)
at io.netty.buffer.PooledByteBufAllocatorL.allocate(PooledByteBufAllocatorL.java:58)
at org.apache.arrow.memory.NettyAllocationManager.<init>(NettyAllocationManager.java:77)
at org.apache.arrow.memory.NettyAllocationManager.<init>(NettyAllocationManager.java:84)
at org.apache.arrow.memory.NettyAllocationManager$1.create(NettyAllocationManager.java:34)
at org.apache.arrow.memory.BaseAllocator.newAllocationManager(BaseAllocator.java:315)
at org.apache.arrow.memory.BaseAllocator.newAllocationManager(BaseAllocator.java:310)
at org.apache.arrow.memory.BaseAllocator.bufferWithoutReservation(BaseAllocator.java:298)
at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:276)
at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:240)
at org.apache.arrow.vector.ipc.message.MessageSerializer.readMessageBody(MessageSerializer.java:725)
at org.apache.arrow.vector.ipc.message.MessageSerializer.deserializeRecordBatch(MessageSerializer.java:363)
at org.apache.spark.sql.execution.arrow.ArrowConverters$.loadBatch(ArrowConverters.scala:189)
at org.apache.spark.sql.execution.arrow.ArrowConverters$$anon$2.nextBatch(ArrowConverters.scala:165)
at org.apache.spark.sql.execution.arrow.ArrowConverters$$anon$2.<init>(ArrowConverters.scala:144)
at org.apache.spark.sql.execution.arrow.ArrowConverters$.fromBatchIterator(ArrowConverters.scala:143)
at org.apache.spark.sql.execution.arrow.ArrowConverters$.$anonfun$toDataFrame$1(ArrowConverters.scala:203)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:855)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:855)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:136)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:839)
We believe this change in netty-handler is where the integration breaks.: netty/netty@9412424
Request for new version of arrow-memory-netty that will work with latest netty-handler
Component(s)
Java
The text was updated successfully, but these errors were encountered:
westonpace
changed the title
Make arrow-memory-netty work with latest netty-handler.jar (4.1.94)
[Java] Make arrow-memory-netty work with latest netty-handler.jar (4.1.94)
Jul 19, 2023
Hey @mrmadira , a new Arrow v13 release is in process right now that will release arrow-memory-netty v13.0.0, which will be pinned to netty v4.1.94.final.
Describe the enhancement requested
Latest arrow-memory-netty jar fails to work with latest netty-handler.jar with the following exception.
We believe this change in netty-handler is where the integration breaks.: netty/netty@9412424
Request for new version of arrow-memory-netty that will work with latest netty-handler
Component(s)
Java
The text was updated successfully, but these errors were encountered: