You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug, including details regarding any error messages, version, and platform.
Fetching data via Apache Arrow Flight (C++, Python involved) and passing them to Apache DataFusion (Rust) does not work:
Memory pointer from external source (e.g, FFI) is not aligned with the specified scalar type.
Before importing buffer through FFI, please make sure the allocation is aligned.
let align = std::mem::align_of::<T>();let is_aligned = buffer.as_ptr().align_offset(align) == 0;match buffer.deallocation(){Deallocation::Standard(_) => assert!(
is_aligned,
"Memory pointer is not aligned with the specified scalar type"),Deallocation::Custom(_, _) =>
assert!(is_aligned, "Memory pointer from external source (e.g, FFI) is not aligned with the specified scalar type. Before importing buffer through FFI, please make sure the allocation is aligned."),}
In my environment, depending on the CSV column, e.g. c1 (c2) with type i32 (i64), align is 4 (8) while buffer.as_ptr().align_offset(align) is always 3 (7), where arrow-rs requires this to be 0.
Component(s)
FlightRPC
The text was updated successfully, but these errors were encountered:
EnricoMi
changed the title
[FlightRPC] Cannot use flight data with datafusion (Rust)
[FlightRPC] Cannot use flight data with DataFusion (Rust)
Aug 5, 2024
Describe the bug, including details regarding any error messages, version, and platform.
Fetching data via Apache Arrow Flight (C++, Python involved) and passing them to Apache DataFusion (Rust) does not work:
This is likely due to #32276 / #36400.
Error:
Reproduce as follows:
git clone --depth=1 https://github.com/apache/arrow.git git clone --depth=1 https://github.com/apache/arrow-testing.git python -m venv venv source venv/bin/activate pip install pyarrow pandas datafusion python arrow/python/examples/flight/server.py RUST_BACKTRACE=1 python example.py arrow-testing/data/csv/aggregate_test_100.csv
with example.py:
The error is thrown in Apache Arrow Rust implementaton: https://github.com/apache/arrow-rs/blob/eddef43d1cb46c1287da187ea1d86b0e1dc35a13/arrow-buffer/src/buffer/scalar.rs#L138
In my environment, depending on the CSV column, e.g.
c1
(c2
) with typei32
(i64
),align
is4
(8
) whilebuffer.as_ptr().align_offset(align)
is always3
(7
), wherearrow-rs
requires this to be0
.Component(s)
FlightRPC
The text was updated successfully, but these errors were encountered: