[Java] Memory allocation alignment

[viirya]

Describe the bug, including details regarding any error messages, version, and platform.

We see memory alignment error when passing arrays allocated using Java arrow to Rust arrow. This is due to strict memory alignment requirement (apache/arrow-rs#4431) at Rust side. I have described the issue there.

So as it looks like it is impossible to access unaligned memory pointers at Rust side. I'm wondering if we could change Java arrow on memory allocation? Could we allocate memory with alignment for arrays in Java arrow so it could be safely sent to other arrow ecosystems?

Component(s)

Java

[lidavidm]

I think it'd be possible & reasonable to (add a mode where) over-allocate by 64 bytes and then use the aligned subset of the buffer. That's what Arrow C++ actually does when using libc malloc.

I'd have to look more deeply into Netty to see what it does though.

Was your Java application using the Netty allocator or the "unsafe" allocator?

[viirya]

We use arrow-memory-unsafe so it is unsafe allocator.

[lidavidm]

CC @davisusanibar

[viirya]

Note that we will probably rely on new arrow-rs API to do re-align for unaligned buffers imported from Java Arrow. So this might be not a blocker for us anymore. But as there will be some overhead when re-aligning buffers, ideally I think it would be great if Java Arrow can do memory alignment.

[lidavidm]

Ping @davisusanibar