Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Java] arrow-vector 16.1.0 has a change that breaks Java 8 support #41717

Closed
pjfanning opened this issue May 19, 2024 · 6 comments
Closed

[Java] arrow-vector 16.1.0 has a change that breaks Java 8 support #41717

pjfanning opened this issue May 19, 2024 · 6 comments

Comments

@pjfanning
Copy link
Contributor

Describe the bug, including details regarding any error messages, version, and platform.

arrow-vector 16.0.0 works fine for us but upgrading to 16.1.0 causes compile issues for us when we build with Java 8.

We found this in the Apache Pekko CI build.

[info] org.apache.pekko.stream.connectors.googlecloud.bigquery.storage.scaladsl.BigQueryArrowStorageSpec *** ABORTED *** (0 milliseconds)
[info]   java.lang.NoSuchMethodError: java.nio.ByteBuffer.rewind()Ljava/nio/ByteBuffer;
[info]   at org.apache.arrow.vector.ipc.message.MessageSerializer.readMessage(MessageSerializer.java:704)
[info]   at org.apache.arrow.vector.ipc.message.MessageSerializer.deserializeSchema(MessageSerializer.java:213)
[info]   at org.apache.pekko.stream.connectors.googlecloud.bigquery.storage.impl.SimpleRowReader.<init>(ArrowSource.scala:66)
[info]   at org.apache.pekko.stream.connectors.googlecloud.bigquery.storage.scaladsl.BigQueryArrowStorageSpec.$anonfun$new$1(BigQueryArrowStorageSpec.scala:35)

Component(s)

Java

@pjfanning
Copy link
Contributor Author

pjfanning commented May 19, 2024

ByteBuffer inherits rewind from Buffer and Java 8 has this method.

https://docs.oracle.com/javase/8/docs/api/java/nio/Buffer.html#rewind--

I have seem issues in Java 8 where you need to cast to java.nio.Buffer for this to work reliably.

In https://issues.apache.org/jira/browse/HADOOP-18197, there were issues where we got on some Java8 installations.

java.lang.NoSuchMethodError: java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;

We had to upgrade protobuf-java (which had the ByteBuffer code that was causing us issues) to get this fix:
protocolbuffers/protobuf@d40aadf

See https://www.morling.dev/blog/bytebuffer-and-the-dreaded-nosuchmethoderror/

@pjfanning
Copy link
Contributor Author

Arrow has this cast in a different part of the code.

((ByteBuffer) ((Buffer) reader).rewind()).get(bytes);

Could we add a similar cast in the method that is causing issues?

@amoeba amoeba changed the title arrow-vector 16.1.0 has a change that breaks Java 8 support [Java] arrow-vector 16.1.0 has a change that breaks Java 8 support May 19, 2024
@assignUser
Copy link
Member

fyi: #38051

pan3793 added a commit to apache/kyuubi that referenced this issue May 20, 2024
# 🔍 Description

A regular dependency upgrading, additionally, Arrow 15 introduced the eclipse-collections dependencies but removed in Arrow 16.

apache/arrow#40896

Note: This PR upgrades Arrow to 16.0.0 instead of 16.1.0 due to apache/arrow#41717

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

Pass GHA

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6391 from pan3793/arrow-16.

Closes #6391

d8ea702 [Cheng Pan] 16.0.0
8a8bc46 [Cheng Pan] Bump Arrow from 15.0.2 to 16.1.0

Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
lidavidm pushed a commit that referenced this issue May 20, 2024
…erializer (#41718)

### Rationale for this change

### What changes are included in this PR?

#41717 describes issue and change

### Are these changes tested?

CI build

### Are there any user-facing changes?

* GitHub Issue: #41717

Authored-by: PJ Fanning <[email protected]>
Signed-off-by: David Li <[email protected]>
@lidavidm lidavidm added this to the 17.0.0 milestone May 20, 2024
@lidavidm
Copy link
Member

Issue resolved by pull request 41718
#41718

@pjfanning
Copy link
Contributor Author

pjfanning commented May 20, 2024

@lidavidm @assignUser This was also reported (@pan3793)

#41718 (comment)

@lidavidm
Copy link
Member

@vibhatha ^

vibhatha pushed a commit to vibhatha/arrow that referenced this issue May 25, 2024
…ssageSerializer (apache#41718)

### Rationale for this change

### What changes are included in this PR?

apache#41717 describes issue and change

### Are these changes tested?

CI build

### Are there any user-facing changes?

* GitHub Issue: apache#41717

Authored-by: PJ Fanning <[email protected]>
Signed-off-by: David Li <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants