Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ray shuffle not working #10

Closed
austin362667 opened this issue Oct 2, 2024 · 2 comments · Fixed by #20
Closed

Ray shuffle not working #10

austin362667 opened this issue Oct 2, 2024 · 2 comments · Fixed by #20

Comments

@austin362667
Copy link
Contributor

I got an error on Mac: Memory pointer from external source (e.g, FFI) is not aligned with the specified scalar type. Before importing buffer through FFI, please make sure the allocation is aligned. while using ray shuffle on Datafusion Ray against TPC-H.

After that the Ray worker died.

@andygrove saw this error when trying to run the example on Linux, but it worked on Mac.

@franklsf95 Any suggestion would be helpful! I appreciate it!
@andygrove
Copy link
Member

Possible workaround in apache/arrow#43552

@andygrove
Copy link
Member

arrow-rs already has a fix for this in apache/arrow-rs#6472 and is included in 53.1.0, so this should be fixed when we upgrade to a version of DataFusion that uses this Arrow version

This was referenced Oct 4, 2024
Open
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix FFI errors, and add script for running TPC-H andygrove/datafusion-ray
2 participants
@andygrove @austin362667