[RLlib] auto API reference pages: rllib/env package only.

doc/source/index.rst

@@ -332,7 +332,7 @@ Papers
    rllib-offline.rst
    rllib-concepts.rst
    rllib-examples.rst
-   rllib-package-ref.rst
+   rllib/package_ref/index.rst
    rllib-dev.rst
 
 .. toctree::

doc/source/rllib-package-ref.rst

@@ -1,40 +1,31 @@
 .. include:: rllib/we_are_hiring.rst
 
-RLlib Package Reference
-=======================
-
 ray.rllib.policy
-----------------
+================
 
 .. automodule:: ray.rllib.policy
     :members:
 
-ray.rllib.env
--------------
-
-.. automodule:: ray.rllib.env
-    :members:
-
 ray.rllib.evaluation
---------------------
+====================
 
 .. automodule:: ray.rllib.evaluation
     :members:
 
 ray.rllib.execution
---------------------
+===================
 
 .. automodule:: ray.rllib.execution
     :members:
 
 ray.rllib.models
-----------------
+================
 
 .. automodule:: ray.rllib.models
     :members:
 
 ray.rllib.utils
----------------
+===============
 
 .. automodule:: ray.rllib.utils
     :members:

doc/source/rllib/__WIP__.txt

@@ -0,0 +1,5 @@
+This folder is WIP and will eventually contain all the revamped documentation source
+files.
+
+Some rst files for the new RLlib documentation have already moved into
+here, others are still residing directly under the main doc/source directory.

doc/source/rllib/package_ref/env.rst

@@ -0,0 +1,47 @@
+.. _env-reference-docs:
+
+Environment APIs
+================
+
+.. toctree::
+   :maxdepth: 1
+
+   env/base_env.rst
+   env/multi_agent_env.rst
+   env/vector_env.rst
+   env/external_env.rst
+
+Any environment type provided by you to RLlib (e.g. a user-defined ``gym.Env`` class),
+is converted internally into the ``BaseEnv`` API, whose main methods are ``poll()`` and ``send_actions()``:
+
+.. https://docs.google.com/drawings/d/1NtbVk-Mo89liTRx-sHu_7fqi3Kn7Hjdf3i6jIMbxGlY/edit
+.. image:: ../../images/rllib/env_classes_overview.svg
+
+
+The ``BaseEnv`` API allows RLlib to support:
+
+1) Vectorization of sub-environments (i.e. individual ``gym.Env`` instances, stacked to form a vector of envs) in order to batch the action computing model forward passes.
+2) External simulators requiring async execution (e.g. envs that run on separate machines and independently request actions from a policy server).
+3) Stepping through the individual sub-environments in parallel via pre-converting them into separate @ray.remote actors.
+4) Multi-agent RL via dicts mapping agent IDs to observations/rewards/etc..
+
+For example, if you provide a custom ``gym.Env`` class to RLlib, auto-conversion to ``BaseEnv`` goes as follows:
+
+- User provides a ``gym.Env`` -> ``_VectorizedGymEnv`` (is-a ``VectorEnv``) -> ``BaseEnv``
+
+Here is a simple example:
+
+.. literalinclude:: ../../../../rllib/examples/documentation/custom_gym_env.py
+   :language: python
+
+..   start-after: __sphinx_doc_model_construct_1_begin__
+..   end-before: __sphinx_doc_model_construct_1_end__
+
+However, you may also conveniently sub-class any of the other supported RLlib-specific
+environment types. The automated paths from those env types (or callables returning instances of those types) to
+an RLlib ``BaseEnv`` is as follows:
+
+- User provides a custom ``MultiAgentEnv`` (is-a ``gym.Env``) -> ``VectorEnv`` -> ``BaseEnv``
+- User uses a policy client (via an external simulator) -> ``ExternalEnv|ExternalMultiAgentEnv`` -> ``BaseEnv``
+- User provides a custom ``VectorEnv`` -> ``BaseEnv``
+- User provides a custom ``BaseEnv`` -> do nothing

doc/source/rllib/package_ref/env/base_env.rst

@@ -0,0 +1,10 @@
+.. _base-env-reference-docs:
+
+BaseEnv API
+===========
+
+rllib.env.base_env.BaseEnv
+--------------------------
+
+.. autoclass:: ray.rllib.env.base_env.BaseEnv
+    :members:

doc/source/rllib/package_ref/env/external_env.rst

@@ -0,0 +1,28 @@
+.. _external-env-reference-docs:
+
+ExternalEnv API
+===============
+
+ExternalEnv (Single-Agent Case)
+-------------------------------
+
+rllib.env.external_env.ExternalEnv
+++++++++++++++++++++++++++++++++++
+
+.. autoclass:: ray.rllib.env.external_env.ExternalEnv
+    :special-members: __init__
+    :members:
+
+
+ExternalMultiAgentEnv (Multi-Agent Case)
+----------------------------------------
+
+rllib.env.external_multi_agent_env.ExternalMultiAgentEnv
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+
+If your external environment needs to support multi-agent RL, you should instead
+sub-class ``ExternalMultiAgentEnv``:
+
+.. autoclass:: ray.rllib.env.external_multi_agent_env.ExternalMultiAgentEnv
+    :special-members: __init__
+    :members:

doc/source/rllib/package_ref/env/multi_agent_env.rst

@@ -0,0 +1,22 @@
+.. _multi-agent-env-reference-docs:
+
+MultiAgentEnv API
+=================
+
+rllib.env.multi_agent_env.MultiAgentEnv
+---------------------------------------
+
+.. autoclass:: ray.rllib.env.multi_agent_env.MultiAgentEnv
+
+    .. automethod:: __init__
+    .. automethod:: reset
+    .. automethod:: step
+    .. automethod:: render
+    .. automethod:: with_agent_groups
+
+
+Convert gym.Env into MultiAgentEnv
+----------------------------------
+
+.. automodule:: ray.rllib.env.multi_agent_env
+    :members: make_multi_agent

doc/source/rllib/package_ref/env/vector_env.rst

@@ -0,0 +1,22 @@
+.. _vector-env-reference-docs:
+
+VectorEnv API
+=============
+
+rllib.env.vector_env.VectorEnv
+------------------------------
+
+.. autoclass:: ray.rllib.env.vector_env.VectorEnv
+    :special-members: __init__
+    :members:
+
+
+Gym.Env to VectorEnv
+--------------------
+
+Internally, RLlib uses the following wrapper class to convert your provided ``gym.Env`` class into
+a ``VectorEnv`` first. After that, RLlib will convert the resulting objects into a ``BaseEnv``.
+
+.. autoclass:: ray.rllib.env.vector_env._VectorizedGymEnv
+    :special-members: __init__
+    :members:

doc/source/rllib/package_ref/index.rst

@@ -0,0 +1,21 @@
+.. include:: ../we_are_hiring.rst
+
+.. _rllib-reference-docs:
+
+RLlib API Reference
+===================
+
+.. tip:: We'd love to hear your feedback on using RLlib - `sign up to our forum and start asking questions <https://discuss.ray.io>`_!
+
+This section contains an overview of RLlib's package- and API reference.
+If you think there is anything missing, please open an issue on `Github`_.
+
+.. _`GitHub`: https://github.com/ray-project/ray/issues
+
+.. TODO: Eventually, move all reference pages into this dir here (doc/source/rllib/package_ref/)
+
+.. toctree::
+   :maxdepth: 2
+
+   env.rst
+   ../../rllib-package-ref