[RLlib; Docs] Docs API reference pages: `rllib/execution`, `rllib/evaluation`, `rllib/models`, `rllib/offline`. #20538

sven1977 · 2021-11-18T15:16:19Z

Docs API reference pages: rllib/execution, rllib/evaluation, rllib/models, rllib/offline.

In a follow up PR: rllib/utils (this would then complete the API reference pages overhaul).

Why are these changes needed?

Related issue number

Checks

I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

…_api_reference_pages_1

…_api_reference_pages_2 # Conflicts: # doc/source/rllib-package-ref.rst # doc/source/rllib/package_ref/index.rst

…es_3

…_api_reference_pages_3

…_api_reference_pages_2

…_api_reference_pages_3

…es_3

…_api_reference_pages_3

sven1977 · 2021-12-01T13:21:38Z

This is ready for review now. @richardliaw
Thanks!

…_api_reference_pages_3 # Conflicts: # rllib/models/modelv2.py

richardliaw · 2021-12-07T04:44:21Z

doc/source/rllib/package_ref/models.rst

+
+All RLlib neural network models have to be provided as ModelV2 sub-classes.
+
+.. autoclass:: ray.rllib.models.modelv2.ModelV2


btw, this isn't showing up properly:

classray.rllib.models.modelv2.ModelV2(obs_space: , action_space: , num_outputs: int, model_config: dict, name: str, framework: str)[source]

This is an existing problem (not introduced by this PR) with gym.spaces.Space, it seems. See e.g. my screenshot below. I'm not sure how to fix this. Sphinx replaces the gym.spaces.Space typehint with some <Mock name='mock.Space' id='...'>-tag ?? (could be because of the string "Space" having a special meaning for Sphinx?).

richardliaw · 2021-12-07T04:45:14Z

doc/source/rllib/package_ref/models.rst

+.. autoclass:: ray.rllib.models.tf.tf_modelv2.TFModelV2
+    :members:
+
+TorchModelV2 (rllib.env.models.torch.torch_modelv2.TorchModelV2)


the name here doesn't render properly, i would suggest removing the location string?

Looks ok to me. I am getting weird text for the gym.spaces.Space typehints, like:

btw I was talking about the sidebar.

Also we have to fix the mock spaces

richardliaw · 2021-12-07T04:45:58Z

doc/source/rllib/package_ref/offline.rst

+InputReaders
+------------
+
+The InputReader API is used by individual :py:class:`~ray.rllib.evaluation.rollout_worker.RolloutWorker`s


didn't render this properly:

richardliaw · 2021-12-07T04:46:35Z

doc/source/rllib/package_ref/offline.rst

+you can use the built-in :py:class:`~ray.rllib.offline.json_reader.JsonReader` class.
+You will have to change the ``input`` config setting from "sampler" (default) to a JSON file name (str), a list of JSON files,
+or a path name (str) that contains json files.
+Alternatively, you can specify a callable that returns a new :py:class:`~ray.rllib.offline.input_reader.InputReader`


show an example?

richardliaw · 2021-12-07T04:47:05Z

doc/source/rllib/package_ref/offline.rst

+       "s3://bucket/expert.json": 0.2,
+    }
+
+.. autoclass:: ray.rllib.offline.mixed_input.MixedInput


I think you are missing a reference to IOContext in the docs.

Yeah, good catch. Will add. ...

richardliaw · 2021-12-07T04:47:34Z

rllib/policy/sample_batch.py


        Returns:
-            MultiAgentBatch: This very MultiAgentBatch.
+            This very MultiAgentBatch.


richardliaw · 2021-12-07T04:47:45Z

rllib/policy/sample_batch.py

@@ -342,11 +338,11 @@ def columns(self, keys: List[str]) -> List[any]:
        return out

    @PublicAPI
-    def shuffle(self) -> None:
+    def shuffle(self) -> "SampleBatch":


nice cleanup!

richardliaw · 2021-12-07T04:48:11Z

doc/source/rllib/package_ref/evaluation.rst

+.. https://docs.google.com/drawings/d/1OewMLAu6KZNon7zpDfZnTh9qiT6m-3M9wnkqWkQQMRc/edit
+.. figure:: ../../images/rllib/rollout_worker_class_overview.svg
+    :align: left
+


this is too big! can you scale it down?

:width: 600

richardliaw · 2021-12-07T04:48:29Z

doc/source/rllib/package_ref/evaluation.rst

+    :align: left
+
+    **A typical RLlib WorkerSet setup inside an RLlib Trainer:** Each ``WorkerSet`` contains
+    exactly one local ``RolloutWorker`` object and n @ray.remote ``RolloutWorkers`` (ray actors).


Suggested change

exactly one local ``RolloutWorker`` object and n @ray.remote ``RolloutWorkers`` (ray actors).

exactly one local ``RolloutWorker`` object and n ray remote ``RolloutWorkers`` (ray actors).

richardliaw · 2021-12-07T04:48:46Z

doc/source/rllib/package_ref/evaluation.rst

+    **A typical RLlib WorkerSet setup inside an RLlib Trainer:** Each ``WorkerSet`` contains
+    exactly one local ``RolloutWorker`` object and n @ray.remote ``RolloutWorkers`` (ray actors).
+    The workers contain a policy map (with one or more policies), and - in case a simulator
+    (env) is available - a vectorized BaseEnv (containing m sub-environments) and


link out to BaseEnv?

Links to all mentioned classes added.

richardliaw · 2021-12-07T04:49:32Z

doc/source/rllib/package_ref/evaluation/rollout_worker.rst

+.. autoclass:: ray.rllib.evaluation.rollout_worker.RolloutWorker
+    :special-members: __init__


man I would consider hiding the docstring of the constructor; it's super long and not intended for users i think

I feel like it's important to keep it for consistency. Users may ask: Why is the init docstring visible in the reference pages for e.g. TorchPolicy, but not for RolloutWorker? Like what's the rule?

…_api_reference_pages_3

sven1977 added 12 commits November 17, 2021 19:37

wip.

645321f

wip.

0c1e393

wip.

270ce56

wip.

d480a79

Merge branch 'master' of https://github.com/ray-project/ray into docs…

d880d17

…_api_reference_pages_1

wip.

90f2498

wip.

03dd8b3

Merge branch 'master' of https://github.com/ray-project/ray into docs…

514bb5a

…_api_reference_pages_2 # Conflicts: # doc/source/rllib-package-ref.rst # doc/source/rllib/package_ref/index.rst

wip.

0c6c80c

Merge branch 'docs_api_reference_pages_2' into docs_api_reference_pag…

addd41d

…es_3

Merge branch 'master' of https://github.com/ray-project/ray into docs…

8a1c122

…_api_reference_pages_3

fix.

62b4c9a

sven1977 added the tests-ok The tagger certifies test failures are unrelated and assumes personal liability. label Nov 19, 2021

sven1977 requested a review from richardliaw November 19, 2021 16:41

sven1977 assigned richardliaw Nov 19, 2021

sven1977 added 12 commits November 22, 2021 10:25

wip.

ea78b56

Merge branch 'master' of https://github.com/ray-project/ray into docs…

1bd213d

…_api_reference_pages_2

wip.

0e0a0d7

Merge branch 'master' of https://github.com/ray-project/ray into docs…

85ac5ef

…_api_reference_pages_2

Merge branch 'master' of https://github.com/ray-project/ray into docs…

d13bead

…_api_reference_pages_2

wip.

5ccee70

Merge branch 'master' of https://github.com/ray-project/ray into docs…

1d650df

…_api_reference_pages_2

wip.

7784da4

Merge branch 'master' of https://github.com/ray-project/ray into docs…

87a8a74

…_api_reference_pages_3

Merge branch 'docs_api_reference_pages_2' into docs_api_reference_pag…

0b495d1

…es_3

wip.

4316ca1

wip.

81bef26

sven1977 changed the title ~~[RLlib; Docs] Docs API reference pages: rllib/execution, rllib/evaluation, rllib/models, rllib/offline.~~ [WIP!! RLlib; Docs] Docs API reference pages: rllib/execution, rllib/evaluation, rllib/models, rllib/offline. Nov 26, 2021

sven1977 added 2 commits December 1, 2021 14:08

Merge branch 'master' of https://github.com/ray-project/ray into docs…

0e83f4c

…_api_reference_pages_3

wip

d37ee78

sven1977 changed the title ~~[WIP!! RLlib; Docs] Docs API reference pages: rllib/execution, rllib/evaluation, rllib/models, rllib/offline.~~ [RLlib; Docs] Docs API reference pages: rllib/execution, rllib/evaluation, rllib/models, rllib/offline. Dec 3, 2021

Merge branch 'master' of https://github.com/ray-project/ray into docs…

50d5baf

…_api_reference_pages_3 # Conflicts: # rllib/models/modelv2.py

richardliaw reviewed Dec 7, 2021

View reviewed changes

sven1977 added 4 commits December 9, 2021 16:19

wip.

e4351d4

wip.

d2db4ce

Merge branch 'master' of https://github.com/ray-project/ray into docs…

6702318

…_api_reference_pages_3

LINT.

736aeb1

richardliaw approved these changes Dec 10, 2021

View reviewed changes

sven1977 merged commit f814c2a into ray-project:master Dec 10, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RLlib; Docs] Docs API reference pages: `rllib/execution`, `rllib/evaluation`, `rllib/models`, `rllib/offline`. #20538

[RLlib; Docs] Docs API reference pages: `rllib/execution`, `rllib/evaluation`, `rllib/models`, `rllib/offline`. #20538

sven1977 commented Nov 18, 2021 •

edited

Loading

sven1977 commented Dec 1, 2021

richardliaw Dec 7, 2021

sven1977 Dec 9, 2021

richardliaw Dec 7, 2021

sven1977 Dec 9, 2021

richardliaw Dec 10, 2021

richardliaw Dec 7, 2021

sven1977 Dec 9, 2021

richardliaw Dec 7, 2021

sven1977 Dec 9, 2021

richardliaw Dec 7, 2021

sven1977 Dec 9, 2021

sven1977 Dec 9, 2021

richardliaw Dec 7, 2021

sven1977 Dec 9, 2021

richardliaw Dec 7, 2021

richardliaw Dec 7, 2021

sven1977 Dec 9, 2021

sven1977 Dec 9, 2021

richardliaw Dec 7, 2021

sven1977 Dec 9, 2021

richardliaw Dec 7, 2021

sven1977 Dec 9, 2021

richardliaw Dec 7, 2021

sven1977 Dec 9, 2021


		All RLlib neural network models have to be provided as ModelV2 sub-classes.

		.. autoclass:: ray.rllib.models.modelv2.ModelV2

	exactly one local ``RolloutWorker`` object and n @ray.remote ``RolloutWorkers`` (ray actors).
	exactly one local ``RolloutWorker`` object and n ray remote ``RolloutWorkers`` (ray actors).

		.. autoclass:: ray.rllib.evaluation.rollout_worker.RolloutWorker
		:special-members: __init__

[RLlib; Docs] Docs API reference pages: rllib/execution, rllib/evaluation, rllib/models, rllib/offline. #20538

[RLlib; Docs] Docs API reference pages: rllib/execution, rllib/evaluation, rllib/models, rllib/offline. #20538

Conversation

sven1977 commented Nov 18, 2021 • edited Loading

Why are these changes needed?

Related issue number

Checks

sven1977 commented Dec 1, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

[RLlib; Docs] Docs API reference pages: `rllib/execution`, `rllib/evaluation`, `rllib/models`, `rllib/offline`. #20538

[RLlib; Docs] Docs API reference pages: `rllib/execution`, `rllib/evaluation`, `rllib/models`, `rllib/offline`. #20538

sven1977 commented Nov 18, 2021 •

edited

Loading