[Core] PyArrow version is out of date

[spolcyn]

What happened + What you expected to happen

The current PyArrow version is locked at <7, which prevents us from using Ray and a modern PyArrow (e.g., v9) version at the same time: https://github.com/ray-project/ray/blob/master/python/requirements.txt#L61.

We'd expect to be able to use the latest version with Ray.

Versions / Dependencies

Ray 2.0.0

Reproduction script

Attempt to build a conda env with ray[all] and pyarrow=9.0.0

Issue Severity

Low: It annoys or frustrates me.

[clarkzinzow]

Hi @spolcyn, thank you for opening this issue! This upper-bound on Arrow is due to a bug in pickling Arrow data that would make using Arrow 7+ in Ray untenable. I actually have a PR out adding support for Arrow 7+ (fixing this Arrow serialization issue with our own workaround), up to the latest stable release (Arrow 9). We expect this to land for Ray 2.2.

Are there any details you can share about why you require support for Arrow 9? E.g. is there an Arrow feature/bug fix that you need, or do you have a dependency that requires a recent Arrow version?

[spolcyn]

@clarkzinzow Great, glad to see that!

At this moment, we don't have any hard requirements for Arrow 7+ -- primary goal is to make sure the ball is already rolling on the PyArrow support so we don't end up in a situation where there is a hard requirement (from us, from another package, etc.).

[thatcort]

I just hit this pyarrow version problem when I tried to use pyarrow.Table.to_pylist() and discovered it doesn't exist.

[clarkzinzow]

@thatcort @spolcyn Just to note, support for Arrow 7 through Arrow 10 (and Arrow nightly) has been added in Ray nightly, and will be released in Ray 2.2! See this milestone for all issues involved with this effort, and the below individual issues and sub-issues for each bug addressed.

• [Datasets] Add support for using Arrow 7 with Ray (Core/Datasets/AIR). #29992
• [Datasets] Add support for using Arrow 8 with Ray (Core/Datasets/AIR). #29995
• [Datasets] Add support for using Arrow 9 with Ray (Core/Datasets/AIR). #29996
• [Datasets] Add support for using Arrow 10 with Ray (Core/Datasets/AIR). #29997
• [Datasets] Add support for using Arrow nightly with Ray (Core/Datasets/AIR). #29998

[spolcyn]

Thanks for making this happen! Looking forward to Ray 2.2

[thatcort]

Hooray! :)