RayDP-1.6.1

Highlights

• Support Spark 3.3.x, 3.4.x, 3.5.x #382 #395 #397 #411
• Dynamic allocation improvement #396
• Fault tolerance improvement #391
• Support setting custom actor owner when convert spark dataframe to ray dataset #376

Thanks @pang-wu, @minmingzhu, @kira-lin, @harborn, @raviranak, @KiranP-d11, @gptbert, @max-509, @Deegue for their contributions to the release!

RayDP-1.6.0

Highlights

• Support Ray 2.1 – 2.6
• Support Spark 3.1-3.4
• Fix logging: only logs from the driver is printed to the console
• Enable OneCCL as backend for TorchTrainer and TorchEstimator
• Add instructions and Dockerfiles for Ray on K8S
• Add resource affinity scheduling for RayDP executors #366. Thanks to @pang-wu

RayDP-1.5.0

Highlights

• Support Ray 2.1.0 - 2.2.0
• Support Spark 3.1 - 3.3
• XGBoostEstimator API #289
• Support converting Spark Dataframe to Ray dataset in a way that data can be recovered in case of failure. #249

RayDP-0.6.0

Highlights

• Support Ray 1.9.0 - 2.1.0
• Support Spark 3.1 - 3.3
• Spark master node affinity
• Updated PyTorch and Tensorflow Estimator using new Ray Train API

Thanks @KepingYan, @kira-lin, @pang-wu, @carsonwang for their contributions to the release!

RayDP-0.5.0

Highlights

• Support Ray 1.9.0 - 2.0.0
• Support Spark 3.1/3.2
• Hive support
• Ray placement group support
• Support multiple users running RayDP on the same node
• Support fractional resource scheduling
• Updated Estimator API using Ray Dataset and Ray Train
• Support custom Spark location by picking up $SPARK_HOME
• RayDP executor extra class path support
• Support data ownership transfer for conversion from Spark Dataframe to Ray Dataset
• New Colab tutorials

Thanks @Bowen0729, @carsonwang, @hezhaozhao-git, @jjyao, @KepingYan, @kira-lin, @marin-ma, @n1CkS4x0, @pang-wu, @wybryan, @Yard1 for their contributions to the release!

RayDP-0.4.2

RayDP-0.4.2 supports Ray 1.12.0

RayDP-0.4.1

RayDP-0.4.1 supports Ray 1.8.0

RayDP-0.4.0

Highlights

• Support conversion between Spark Dataframe and Ray Dataset
• Support Ray 1.7.0
• Support Spark 3.2.0
• Various bug fixes and improvements

Thanks @ConeyLiu @zuston @kira-lin @mjschock @edoakes @carsonwang for their contributions to the release!

RayDP-0.3.0

RayDP 0.3.0 includes the following key changes:

• Spark dynamic resource allocation support. This allows you to launch Spark external shuffle service on Ray and enable Spark dynamic resource allocation to maximize your resource utilization.
• Spark-submit support. A command line utility bin/raydp-submit is provided for you to submit a scala/java/python Spark application to a Ray cluster.
• MPI on Ray. This allows you to run MPI jobs on Ray. You can use this feature to construct pipelines like Spark + MPI on Ray.
• Ray 1.3.0 support.

RayDP-0.2.0

Several bug fixes. And also we have added some examples to show how RayDP works together with other libraries, such as PyTorch, Tensorflow, XGBoost, and Horovod.