[AIR][Train][Doc] Restructure API reference (#32360)

This PR splits up long API refs in AIR and Train into individual pages, one dedicated to each method/class. This PR is a followup to #31204 and #32311, which made the same changes for Ray Data/Tune docs. Signed-off-by: Justin Yu <[email protected]>
ray-project · Feb 14, 2023 · d87d86f · d87d86f
1 parent d092b12
commit d87d86f
Show file tree

Hide file tree

Showing 20 changed files with 470 additions and 634 deletions.
diff --git a/.buildkite/pipeline.build.yml b/.buildkite/pipeline.build.yml
@@ -574,6 +574,10 @@
     - pushd /ray && git clean -f -f -x -d -e .whl -e python/ray/dashboard/client && popd
     - bazel clean --expunge
     - export WANDB_MODE=offline
+    # Horovod needs to be installed separately (needed for API ref imports)
+    - ./ci/env/install-horovod.sh
+    # See https://stackoverflow.com/questions/63383400/error-cannot-uninstall-ruamel-yaml-while-creating-docker-image-for-azure-ml-a
+    - pip install mosaicml==0.10.1 --ignore-installed
     - ./ci/ci.sh build
 
 - label: ":octopus: Tune multinode tests"

diff --git a/doc/requirements-doc.txt b/doc/requirements-doc.txt
@@ -36,6 +36,9 @@ torch
 torchvision
 transformers
 
+# Train
+mxnet==1.8.0.post0
+
 # Ray libraries
 git+https://github.com/ray-project/tune-sklearn@master#tune-sklearn
 git+https://github.com/ray-project/xgboost_ray@master#egg=xgboost_ray

diff --git a/doc/source/_toc.yml b/doc/source/_toc.yml
@@ -118,7 +118,7 @@ parts:
               - file: ray-air/examples/automl_with_ray_air
               - file: ray-air/examples/batch_tuning
               - file: ray-air/examples/batch_forecasting
-          - file: ray-air/package-ref
+          - file: ray-air/api/api
           - file: ray-air/benchmarks
 
       - file: data/dataset
@@ -181,7 +181,7 @@ parts:
               - file: train/examples/pytorch/torch_data_prefetch_benchmark/benchmark_example
                 title: "Torch Data Prefetching Benchmark"
           - file: train/faq
-          - file: train/api
+          - file: train/api/api
 
       - file: tune/index
         title: Ray Tune

diff --git a/doc/source/ray-air/api/api.rst b/doc/source/ray-air/api/api.rst
@@ -0,0 +1,19 @@
+.. _air-api-ref:
+
+Ray AIR API
+===========
+
+.. toctree::
+    :maxdepth: 2
+
+    Preprocessor (Ray Data + Ray Train)<preprocessor.rst>
+    Dataset Ingest (Ray Data + Ray Train)<dataset-ingest.rst>
+    Trainers (Ray Train)<../../train/api/api>
+    Tuner (Ray Tune)<../../tune/api/execution>
+    Results (Ray Train + Ray Tune)<../../tune/api/result_grid>
+    AIR Session (Ray Train + Ray Tune)<session>
+    AIR Configurations (Ray Train + Ray Tune)<configs.rst>
+    AIR Checkpoint (All Libraries)<checkpoint.rst>
+    Predictors (Ray Data + Ray Train)<predictor.rst>
+    Model Serving in AIR (Ray Serve)<serve.rst>
+    External Library Integrations<../../tune/api/integration>
diff --git a/doc/source/ray-air/api/checkpoint.rst b/doc/source/ray-air/api/checkpoint.rst
@@ -0,0 +1,47 @@
+.. _air-checkpoint-ref:
+
+Ray AIR Checkpoint
+==================
+
+.. seealso::
+
+    See :ref:`this API reference section <train-framework-specific-ckpts>` for
+    framework-specific checkpoints used with AIR's library integrations.
+
+Constructor Options
+-------------------
+
+.. currentmodule:: ray.air.checkpoint
+
+.. autosummary::
+    :toctree: doc/
+
+    Checkpoint
+    Checkpoint.from_dict
+    Checkpoint.from_bytes
+    Checkpoint.from_directory
+    Checkpoint.from_uri
+    Checkpoint.from_checkpoint
+
+Checkpoint Properties
+---------------------
+
+.. autosummary::
+    :toctree: doc/
+
+    Checkpoint.uri
+    Checkpoint.get_internal_representation
+    Checkpoint.get_preprocessor
+    Checkpoint.set_preprocessor
+
+Checkpoint Format Conversions
+-----------------------------
+
+.. autosummary::
+    :toctree: doc/
+
+    Checkpoint.to_dict
+    Checkpoint.to_bytes
+    Checkpoint.to_directory
+    Checkpoint.as_directory
+    Checkpoint.to_uri
diff --git a/doc/source/ray-air/api/configs.rst b/doc/source/ray-air/api/configs.rst
@@ -0,0 +1,25 @@
+
+Ray AIR Configurations
+======================
+
+.. TODO(ml-team): Add a general AIR configuration guide that covers all of these configs.
+
+.. seealso::
+
+    See :ref:`this Ray Train configuration user guide <train-config>` for more details.
+
+.. currentmodule:: ray
+
+.. autosummary::
+    :toctree: doc/
+
+    air.RunConfig
+    air.ScalingConfig
+    air.DatasetConfig
+    air.CheckpointConfig
+    air.FailureConfig
+
+.. autosummary::
+
+    tune.TuneConfig
+    tune.syncer.SyncConfig
diff --git a/doc/source/ray-air/api/dataset-ingest.rst b/doc/source/ray-air/api/dataset-ingest.rst
@@ -0,0 +1,23 @@
+Ray Dataset Ingest into AIR Trainers
+=====================================
+
+.. seealso::
+
+    See this :ref:`AIR Dataset ingest guide <air-configure-ingest>` for usage examples.
+
+.. currentmodule:: ray
+
+.. autosummary::
+
+    air.session.get_dataset_shard
+    ~data.DatasetIterator
+    ~air.DatasetConfig
+
+Debugging Utilities
+-------------------
+
+.. autosummary::
+    :toctree: doc/
+
+    ~air.util.check_ingest.make_local_dataset_iterator
+    ~air.util.check_ingest.DummyTrainer
diff --git a/doc/source/ray-air/api/predictor.rst b/doc/source/ray-air/api/predictor.rst
@@ -0,0 +1,87 @@
+Predictor
+=========
+
+.. seealso::
+
+    See this :ref:`user guide on performing model inference <air-predictors>` in
+    AIR for usage examples.
+
+.. currentmodule:: ray.train
+
+Predictor Interface
+-------------------
+
+Constructor Options
+~~~~~~~~~~~~~~~~~~~
+
+.. autosummary::
+    :toctree: doc/
+
+    predictor.Predictor
+    predictor.Predictor.from_checkpoint
+    predictor.Predictor.from_pandas_udf
+
+Predictor Properties
+~~~~~~~~~~~~~~~~~~~~
+
+.. autosummary::
+    :toctree: doc/
+
+    predictor.Predictor.get_preprocessor
+    predictor.Predictor.set_preprocessor
+
+
+Prediction API
+~~~~~~~~~~~~~~~
+
+.. autosummary::
+    :toctree: doc/
+
+    predictor.Predictor.predict
+
+
+Supported Data Formats
+~~~~~~~~~~~~~~~~~~~~~~
+
+.. autosummary::
+    :toctree: doc/
+
+    predictor.Predictor.preferred_batch_format
+    ~predictor.DataBatchType
+
+
+Batch Predictor
+---------------
+
+Constructor
+~~~~~~~~~~~
+
+.. autosummary::
+    :toctree: doc/
+
+    batch_predictor.BatchPredictor
+
+
+Batch Prediction API
+~~~~~~~~~~~~~~~~~~~~
+
+.. autosummary::
+    :toctree: doc/
+
+    batch_predictor.BatchPredictor.predict
+    batch_predictor.BatchPredictor.predict_pipelined
+
+
+Built-in Predictors for Library Integrations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autosummary::
+    :toctree: doc/
+
+    ~xgboost.XGBoostPredictor
+    ~lightgbm.LightGBMPredictor
+    ~tensorflow.TensorflowPredictor
+    ~torch.TorchPredictor
+    ~huggingface.HuggingFacePredictor
+    ~sklearn.SklearnPredictor
+    ~rl.RLPredictor
diff --git a/doc/source/ray-air/api/preprocessor.rst b/doc/source/ray-air/api/preprocessor.rst
@@ -0,0 +1,94 @@
+.. _air-preprocessor-ref:
+
+Preprocessor
+============
+
+Preprocessor Interface
+------------------------
+
+.. currentmodule:: ray.data
+
+Constructor
+~~~~~~~~~~~
+
+.. autosummary::
+    :toctree: doc/
+
+    ~preprocessor.Preprocessor
+
+Fit/Transform APIs
+~~~~~~~~~~~~~~~~~~
+
+.. autosummary::
+    :toctree: doc/
+
+    ~preprocessor.Preprocessor.fit
+    ~preprocessor.Preprocessor.fit_transform
+    ~preprocessor.Preprocessor.transform
+    ~preprocessor.Preprocessor.transform_batch
+    ~preprocessor.Preprocessor.transform_stats
+
+
+Generic Preprocessors
+---------------------
+
+.. autosummary::
+    :toctree: doc/
+
+    ~preprocessors.BatchMapper
+    ~preprocessors.Chain
+    ~preprocessors.Concatenator
+    ~preprocessors.SimpleImputer
+
+Categorical Encoders
+--------------------
+
+.. autosummary::
+    :toctree: doc/
+
+    ~preprocessors.Categorizer
+    ~preprocessors.LabelEncoder
+    ~preprocessors.MultiHotEncoder
+    ~preprocessors.OneHotEncoder
+    ~preprocessors.OrdinalEncoder
+
+Feature Scalers
+---------------
+
+.. autosummary::
+    :toctree: doc/
+
+    ~preprocessors.MaxAbsScaler
+    ~preprocessors.MinMaxScaler
+    ~preprocessors.Normalizer
+    ~preprocessors.PowerTransformer
+    ~preprocessors.RobustScaler
+    ~preprocessors.StandardScaler
+
+K-Bins Discretizers
+-------------------
+
+.. autosummary::
+    :toctree: doc/
+
+    ~preprocessors.CustomKBinsDiscretizer
+    ~preprocessors.UniformKBinsDiscretizer
+
+Image Preprocessors
+-------------------
+
+.. autosummary::
+    :toctree: doc/
+
+    ~preprocessors.TorchVisionPreprocessor
+
+Text Encoders
+-------------
+
+.. autosummary::
+    :toctree: doc/
+
+    ~preprocessors.CountVectorizer
+    ~preprocessors.FeatureHasher
+    ~preprocessors.HashingVectorizer
+    ~preprocessors.Tokenizer
diff --git a/doc/source/ray-air/api/serve.rst b/doc/source/ray-air/api/serve.rst
@@ -0,0 +1,18 @@
+.. _air-serve-integration:
+
+Model Serving in AIR
+=====================
+
+.. seealso::
+
+    See this :ref:`model serving guide <air-serving-guide>` to see how Ray Serve
+    can be used within the Ray AIR ecosystem.
+
+.. currentmodule:: ray.serve.air_integrations
+
+.. autosummary::
+    :toctree: doc/
+
+    PredictorWrapper
+
+.. autoclass:: PredictorDeployment