[RLlib] Cleanup `examples/` folder 03. #44559

sven1977 · 2024-04-08T08:40:11Z

Cleanup examples/ folder (vol. 03).

The planned, final structure for the examples folder of RLlib will be as follows:

rllib/examples/
    algorithms/ (custom algo examples)
    catalogs/ (catalog examples)
    checkpoints/ (examples on exporting models, algos, env-runners, etc.c.)
    connectors/ (examples for custom ConnectorV2)
    curriculum/ (curriculum learning)
    debugging/ (tips and tricks for debugging)
    envs/ (example scripts for env handling AND example classes)
    evaluation/ (how to setup evaluation workers)
    gpus/ (GPU training)
    hierarchical/ (hierarchical RL)
    inference/ (examples on how to do inference after training)
    learners/ (example scripts for Learner handling AND example Learner classes)
    multi_agent/ (multi-agent RL examples and classes)
    offline_rl/ (offline RL examples)
    ray_serve/ (RLlib with Ray Serve)
    ray_tune/ (RLlib with Ray Tune)
    rl_modules/ (example scripts for RLModule handling AND example custom RLModule classes)

All scripts that are currently sitting in the root examples directory will move into one of the above sub-folders.
Moved scripts that have been part of RLlib for a long time will remain as-is, BUT create error messages pointing the user to the new location. In a few major releases (before 3.0), we will completely remove these stubs.
A temporary sub-folder (_old_api_stack) has been created to host only those scripts that show things that are relevant to the old API stack only. This folder will also eventually disappear (before 3.0).
The current sub-folders: old_api_stack, catalog, env, export, inference_and_serving, learner, policy, rl_module, and serving will all be removed soon (replaced by the plural naming (e.g. env -> envs) or a cleaner name (e.g. export -> checkpoints) or replaced altogether (e.g. policy)).

TODOs (in several follow-up PRs):

Many example scripts (even those already moved into the proper sub-folders) still only work on the old API stack and will have to be "translated" into the new API stack.
Some example scripts - for this PR - remain in the root rllib.examples folder for now until we figure out their final location (e.g. parametric_actions_cartpole.py). Most of these are related to action spaces or action manipulation and will most likely move into the connectors folder, but we'll see.

Why are these changes needed?

Related issue number

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Signed-off-by: sven1977 <[email protected]>

…nup_examples_folder_03 # Conflicts: # rllib/examples/multi_agent/multi_agent_pendulum.py and wip Signed-off-by: sven1977 <[email protected]>

Signed-off-by: sven1977 <[email protected]>

simonsays1980

LGTM. I think that labeling some examples as "only-old-stack" or putting them into a folder "old-stack" helps users to distinguish which examples can be executed in the new stack.

simonsays1980 · 2024-04-08T12:48:17Z

rllib/examples/checkpoints/restore_1_of_n_agents_from_checkpoint.py

+    }
+
+    class RestoreWeightsCallback(DefaultCallbacks):
+        def on_algorithm_init(self, *, algorithm: "Algorithm", **kwargs) -> None:


Is this callback actually called before weights synching?

Great question. I'll check this and fix, if necessary. ...

This callback is called after the initial weights initialization of all workers (from the randomly initialized learners).

So this should be fine.

First we initialize the algo randomly (and sync all the workers).

Then we override only(!) the 1st policy with the given weights.

simonsays1980 · 2024-04-08T12:52:53Z

rllib/examples/envs/classes/multi_agent.py

+)
+
+
+class GuessTheNumberGame(MultiAgentEnv):


Do we have a multi-agent environment that uses __common__ infos?

Good question. We should make this a separate (new) example script, then.

simonsays1980 · 2024-04-08T12:56:33Z

rllib/examples/envs/unity3d_env_local.py

@@ -0,0 +1,207 @@
+# TODO (sven): Move this example script into the new API stack.


Afaics this example is only for old stack (using exploration). Can we put old stack examples under a specific folder old stack?

I see users trying to use exploration algorithms with our new stack and failing with this combination.

Well, we'll have to translate it into the new stack (using RLModule instead of the old Exploration API).

simonsays1980 · 2024-04-08T12:58:48Z

rllib/examples/gpus/fractional_gpus.py

+        )
+        .framework(args.framework)
+        .resources(
+            # How many GPUs does the local worker (driver) need? For most algos,


Can we put here also an example for mutiple worker learners? This is still ambiguous how exactly to set multiple learner workers and assign them GPUs.

Added a TODO. Sorry, I'm trying to keep this PR completely free of actual code changes.

simonsays1980 · 2024-04-08T13:02:18Z

rllib/examples/multi_agent/two_algorithms.py

@@ -0,0 +1,188 @@
+# TODO (sven): Move this example script into the new API stack.


Here as well: Is this possible with new stack and can we otherwise put this example under a specific foler named old_stack?

There is a new folder called _old_api_stack, but I would like to only put scripts in there that don't need to be translated to the new API stack b/c whatever is demo'd in these will no longer be supported (or relevant).

All the other scripts currently still on the old stack (that are NOT in that folder) will have to be translated.

I agree with you that we should mark all these differences more clearly.
Suggestion:

Old-API stack scripts that will NOT be supported in the future (those in _old_api_stack/): add deprecation_warning to top explaining that these scripts will be deprecated fully (w/o replacement)

Old-API stack scripts that we will translated (those NOT in _old_api_stack/): add warning to top explaining that we are about to translate these to the new stack (and will retire/remove their old stack version).

New API stack: Leave as-is (this should be the new normal).

Yes, it should be possible. This is not even new stack specific. We just have to use PPO/DQNRLModule instead of PPO/DQNPolicy.

Signed-off-by: sven1977 <[email protected]>

…nup_examples_folder_03

doc/source/rllib/rllib-env.rst

doc/source/rllib/rllib-models.rst

Signed-off-by: sven1977 <[email protected]>

Co-authored-by: angelinalg <[email protected]> Signed-off-by: Sven Mika <[email protected]>

Signed-off-by: sven1977 <[email protected]>

… cleanup_examples_folder_03

Signed-off-by: sven1977 <[email protected]>

…nup_examples_folder_03

sven1977 added 4 commits April 6, 2024 20:33

wip

bdaa04c

Signed-off-by: sven1977 <[email protected]>

wip

52d9e12

Signed-off-by: sven1977 <[email protected]>

wip

f77ffdb

Signed-off-by: sven1977 <[email protected]>

Merge branch 'master' of https://github.com/ray-project/ray into clea…

81c4c79

…nup_examples_folder_03 # Conflicts: # rllib/examples/multi_agent/multi_agent_pendulum.py and wip Signed-off-by: sven1977 <[email protected]>

sven1977 requested review from avnishn, ArturNiederfahrenhorst, maxpumperla, kouroshHakha and simonsays1980 as code owners April 8, 2024 08:40

sven1977 assigned simonsays1980 and angelinalg Apr 8, 2024

sven1977 added 2 commits April 8, 2024 11:55

wip

1672675

Signed-off-by: sven1977 <[email protected]>

wip

adf9e8c

Signed-off-by: sven1977 <[email protected]>

sven1977 requested a review from a team as a code owner April 8, 2024 10:59

wip

c9e5c2f

Signed-off-by: sven1977 <[email protected]>

simonsays1980 approved these changes Apr 8, 2024

View reviewed changes

sven1977 added 9 commits April 8, 2024 17:07

wip

683bc4b

Signed-off-by: sven1977 <[email protected]>

wip

5bd220f

Signed-off-by: sven1977 <[email protected]>

LINT

d931945

Signed-off-by: sven1977 <[email protected]>

wip

e9888de

Signed-off-by: sven1977 <[email protected]>

fixes

0e97d8f

Signed-off-by: sven1977 <[email protected]>

fixes

7584cce

Signed-off-by: sven1977 <[email protected]>

fixes

5565d4f

Signed-off-by: sven1977 <[email protected]>

fixes

27c793d

Signed-off-by: sven1977 <[email protected]>

Merge branch 'master' of https://github.com/ray-project/ray into clea…

d75b31a

…nup_examples_folder_03

angelinalg approved these changes Apr 8, 2024

View reviewed changes

doc/source/rllib/rllib-env.rst Outdated Show resolved Hide resolved

doc/source/rllib/rllib-models.rst Outdated Show resolved Hide resolved

doc/source/rllib/rllib-models.rst Outdated Show resolved Hide resolved

sven1977 and others added 4 commits April 9, 2024 07:25

fixes

bf9cef0

Signed-off-by: sven1977 <[email protected]>

Apply suggestions from code review

872b49b

Co-authored-by: angelinalg <[email protected]> Signed-off-by: Sven Mika <[email protected]>

fixes

743dabd

Signed-off-by: sven1977 <[email protected]>

Merge remote-tracking branch 'origin/cleanup_examples_folder_03' into…

d50a39c

… cleanup_examples_folder_03

sven1977 added 2 commits April 9, 2024 10:26

fix

d1a18c6

Signed-off-by: sven1977 <[email protected]>

Merge branch 'master' of https://github.com/ray-project/ray into clea…

a092ea3

…nup_examples_folder_03

sven1977 merged commit 1d69833 into ray-project:master Apr 9, 2024
4 of 5 checks passed

sven1977 deleted the cleanup_examples_folder_03 branch April 9, 2024 10:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RLlib] Cleanup `examples/` folder 03. #44559

[RLlib] Cleanup `examples/` folder 03. #44559

sven1977 commented Apr 8, 2024 •

edited

Loading

simonsays1980 left a comment

simonsays1980 Apr 8, 2024

sven1977 Apr 8, 2024

sven1977 Apr 8, 2024

sven1977 Apr 8, 2024

simonsays1980 Apr 8, 2024

sven1977 Apr 8, 2024

simonsays1980 Apr 8, 2024

sven1977 Apr 8, 2024

simonsays1980 Apr 8, 2024

sven1977 Apr 8, 2024

simonsays1980 Apr 8, 2024

sven1977 Apr 8, 2024

sven1977 Apr 8, 2024

		@@ -0,0 +1,207 @@
		# TODO (sven): Move this example script into the new API stack.

		@@ -0,0 +1,188 @@
		# TODO (sven): Move this example script into the new API stack.

[RLlib] Cleanup examples/ folder 03. #44559

[RLlib] Cleanup examples/ folder 03. #44559

Conversation

sven1977 commented Apr 8, 2024 • edited Loading

Why are these changes needed?

Related issue number

Checks

simonsays1980 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

[RLlib] Cleanup `examples/` folder 03. #44559

[RLlib] Cleanup `examples/` folder 03. #44559

sven1977 commented Apr 8, 2024 •

edited

Loading