[Datasets] Add support for using Arrow 10 with Ray (Core/Datasets/AIR).

[clarkzinzow]

Arrow 10 has some breaking API changes and bugs that we need to accommodate in order to make it usable with Ray.

Known Issues

• Everything for Arrow 9 support: [Datasets] Add support for using Arrow 9 with Ray (Core/Datasets/AIR). #29996
• [Datasets] In Arrow 10+, S3 errors raised due to permission issues can vary beyond our current pattern matching #29994

Success Criteria

• CI job running against Arrow 10 is passing

[thatcort]

I've tried including both ray[air]==2.2.0 and pyarrow==10.0.1 in the same requirements.txt file but get the following error:

ERROR: Cannot install pyarrow==10.0.1 and ray[air]==2.2.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested pyarrow==10.0.1
    ray[air] 2.2.0 depends on pyarrow<8.0.0 and >=6.0.1; extra == "air"

[clarkzinzow]

Hi @thatcort, unfortunately removal of the pyarrow<8.0.0 upperbound was missed in this PR, and the subsequent hotfix missed the Ray 2.2 release.

This will work as expected in the upcoming Ray 2.3 release. but if you're looking to unblock your workload, you could either:

1. Use a nightly Ray wheel.
2. Do an override install of pyarrow after installling ray:

pip install ray[air]==2.2.0
pip install pyarrow==10.0.1