Ray-2.35.0

Notice: Starting from this release, pip install ray[all] will not include ray[cpp], and will not install the respective ray-cpp package. To install everything that includes ray-cpp, one can use pip install ray[cpp-all] instead.

Ray Libraries

Ray Data

🎉 New Features:

• Upgrade supported Arrow version from 16 to 17 (#47034)
• Add support for reading from Iceberg (#46889)

💫 Enhancements:

• Various Progress Bar UX improvements (#46816, #46801, #46826, #46692, #46699, #46974, #46928, #47029, #46924, #47120, #47095, #47106)
• Try get size_bytes from metadata and consolidate metadata methods (#46862)
• Improve warning message when read task is large (#46942)
• Extend API to enable passing sample weights via ray.dataset.to_tf (#45701)
• Add a parameter to allow overriding LanceDB scanner options (#46975)
• Add failure retry logic for read_lance (#46976)
• Clarify warning for reading old Parquet data (#47049)
• Move datasource implementations to _internal subpackage (#46825)
• Handle logs from tensor extensions (#46943)

🔨 Fixes:

• Change type of DataContext.retried_io_errors from tuple to list (#46884)
• Make Parquet tests more robust and expose Parquet logic (#46944)
• Change pickling log level from warning to debug (#47032)
• Add validation for shuffle arg (#47055)
• Fix validation bug when size=0 in ActorPoolStrategy (#47072)
• Fix exception in async map (#47110)
• Fix wrong metrics group for Object Store Memory metrics on Ray Data Dashboard (#47170)
• Handle errors in SplitCoordinator when generating a new epoch (#47176)

📖 Documentation:

• Auto-gen GroupedData api (#46925)
• Fix signature of Rule.plan (#47094)

Ray Train

💫 Enhancements:

• [train] Updates to support xgboost==2.1.0 (#46667)
• [train] Add hardware stats (#46719)

Ray Tune

🔨 Fixes:

• [RLlib; Tune] Fix WandB metric overlap after restore from checkpoint. (#46897)

Ray Serve

💫 Enhancements:

• Improved handling of replica death and replica unavailability in deployment handle routers before controller restarts replica (#47008)
• Eagerly create routers in proxy for better GCS fault tolerance (#47031)
• Immediately send ping in router when receiving new replica set (#47053)

🏗 Architecture refactoring:

• Deprecate passing arguments that contain DeploymentResponses in nested objects to downstream deployment handle calls (#46806)

RLlib

🎉 New Features:

• Offline RL on the new API stack:
  • Record offline data (#46818, #47046, #47133, #47155) and support to directly read from episodes. (#46865)
  • RLUnplugged example. (#46792)
  • Progress on BC/MARWIL migration: #44970, #47154, #46799
  • Progress on CQL migration: #46969, #47105

💫 Enhancements:

• Add ObservationPreprocessor (ConnectorV2). (#47077)

🔨 Fixes:

• New API stack: Fix IMPALA/APPO + LSTM for single- and multi-GPU. (#47132, #47158)
• Various bug fixes: #46898, #47047, #46963, #47021, #46897
• Add more control to Algorithm.add_module/policy methods. (#46932, #46836)

📖 Documentation:

• Example scripts for new API stack:
  • Curiosity (inverse dynamics model-based) RLModule example. (#46841)
  • Add example script for Env with protobuf observation space. (#47071)
• New API stack documentation:
  • Cleanup old API stack docs (rllib-dev.rst). (#47172)
  • Episodes (SingleAgentEpisode). (#46985)
  • Redo rllib-algorithms.rst page. (#46916)

🏗 Architecture refactoring:

• Rename MultiAgent...RLModule... into MultiRL...Module for more generality. (#46840)
• Add learner_only flag to RLModuleConfig/Spec and simplify creation of RLModule specs from algo-config. (#46900)

Ray Core

💫 Enhancements:

• Emit total lineage bytes metrics (#46725)
• Adding accelerator type H100 (#46823)
• More structured logging in core worker (#46906)
• Change all callbacks to move to save copies. (#46971)
• Add ray[adag] option to pip install (#47009)

🔨 Fixes:

• Fix dashboard process reporting on windows (#45578)
• Fix Ray-on-Spark cluster crashing bug when user cancels cell execution (#46899)
• Fix PinExistingReturnObject segfault by passing owner_address (#46973)
• Fix raylet CHECK failure from runtime env creation failure. (#46991)
• Fix typo in memray command (#47006)
• [ADAG] Fix for asyncio outputs (#46845)

📖 Documentation:

• Clarify behavior of placement_group_capture_child_tasks in docs (#46885)
• Update ray.available_resources() docstring (#47018)

🏗 Architecture refactoring:

• Async APIs for the New GcsClient. (#46788)
• Replace GCS stubs in the dashboard to use NewGcsAioClient. (#46846)

Dashboard

💫 Enhancements:

• Polish and minor improvements to the Serve page (#46811)

🔨 Fixes:

• Fix CPU/GPU/RAM not being reported correctly on Windows (#44578)

Docs

💫 Enhancements:

• Add more information about developer tooling for docs contributions (#46636), including esbonio section

🔨 Fixes:

• Use PyData Sphinx theme version switcher (#46936)

Thanks

Many thanks to all those who contributed to this release!
@simonsays1980, @bveeramani, @tungh2, @zcin, @xingyu-long, @WeichenXu123, @aslonnie, @MaxVanDijck, @can-anyscale, @galenhwang, @omatthew98, @matthewdeng, @raulchen, @sven1977, @shrekris-anyscale, @deepyaman, @alexeykudinkin, @stephanie-wang, @kevin85421, @ruisearch42, @hongchaodeng, @khluu, @alanwguo, @hongpeng-guo, @saihaj, @Superskyyy, @tespent, @slfan1989, @justinvyu, @rynewang, @nikitavemuri, @amogkam, @mattip, @dev-goyal, @ryanaoleary, @peytondmurray, @edoakes, @venkatajagannath, @jjyao, @cristianjd, @scottjlee, @Bye-legumes