[RLlib] Preparatory PR: Make EnvRunners use (enhanced) Connector API (#01: mostly cleanups and small fixes) #41074

sven1977 · 2023-11-10T14:17:29Z

Preparatory PR: Make EnvRunners use (enhanced) Connector API (#1: mostly cleanups and small fixes)

Why are these changes needed?

Related issue number

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Signed-off-by: sven1977 <[email protected]>

sven1977 · 2023-11-14T11:35:03Z

rllib/algorithms/algorithm.py

@@ -1932,7 +1932,10 @@ def compute_actions(
        filtered_obs, filtered_state = [], []
        for agent_id, ob in observations.items():
            worker = self.workers.local_worker()
-            preprocessed = worker.preprocessors[policy_id].transform(ob)
+            if worker.preprocessors.get(policy_id) is not None:


This is a bug fix.

sven1977 · 2023-11-14T11:35:34Z

rllib/algorithms/algorithm_config.py

@@ -319,26 +319,37 @@ def __init__(self, algo_class=None):
        # If not specified, we will try to auto-detect this.
        self._is_atari = None

+        # TODO (sven): Rename this method into `AlgorithmConfig.sampling()`


Now that we are aiming for a the EnvRunner API as the default, we should rename/clarify some of these config settings and methods.

Please consider loading a checkpoint here? Are these renaming backward compatible?

Is there even a story around this? Like can people even move from rllib 2+ to 3?

sven1977 · 2023-11-14T11:35:56Z

rllib/core/models/torch/encoder.py

@@ -285,30 +285,31 @@ def __init__(self, config: RecurrentEncoderConfig) -> None:
            bias=config.use_bias,
        )

+        self.state_in_out_spec = {


Simplified (repetitive) code.

make this private attribute?

sven1977 · 2023-11-14T11:37:01Z

rllib/env/multi_agent_episode.py

@@ -212,75 +212,75 @@ def get_observations(

        return self._getattr_by_index("observations", indices, global_ts)

-    def get_actions(
+    def get_infos(


Reordered:

obs, infos (<- env.reset data)

action, reward, terminated/truncated (<- other env.step results)

extra model outs

sven1977 · 2023-11-14T11:37:17Z

rllib/env/single_agent_env_runner.py

-        gym.register(
-            "custom-env-v0",
-            partial(
+        if (


sven1977 · 2023-11-14T11:37:43Z

rllib/evaluation/worker_set.py

@@ -690,6 +690,9 @@ def foreach_worker(
        if local_worker and self.local_worker() is not None:
            local_result = [func(self.local_worker())]

+        if not self.__worker_manager.actor_ids():


Shortcut for local-worker only case.

sven1977 · 2023-11-14T11:37:56Z

rllib/tuned_examples/appo/multi-agent-cartpole-crashing-restart-env-appo.yaml

@@ -30,7 +30,7 @@ multi-agent-cartpole-crashing-appo:
        # Switch on resiliency for failed sub environments (within a vectorized stack).
        restart_failed_sub_environments: true

-        # Switch on evaluation workers being managed by AsyncRequestsManager object.
+        # Switch on asynchronous handling of evaluation workers.


AsyncRequestsManager doesn't exist anymore.

sven1977 · 2023-11-14T11:38:20Z

rllib/utils/spaces/space_utils.py

@@ -205,6 +205,56 @@ def flatten_to_single_ndarray(input_):
    return input_


+@DeveloperAPI


Very useful new utility. Inverse of already existing unbatch utility.

kouroshHakha · 2023-11-16T04:26:32Z

rllib/algorithms/algorithm_config.py

@@ -319,26 +319,37 @@ def __init__(self, algo_class=None):
        # If not specified, we will try to auto-detect this.
        self._is_atari = None

+        # TODO (sven): Rename this method into `AlgorithmConfig.sampling()`


Please consider loading a checkpoint here? Are these renaming backward compatible?

kouroshHakha · 2023-11-16T04:28:05Z

rllib/algorithms/algorithm_config.py

@@ -319,26 +319,37 @@ def __init__(self, algo_class=None):
        # If not specified, we will try to auto-detect this.
        self._is_atari = None

+        # TODO (sven): Rename this method into `AlgorithmConfig.sampling()`


Is there even a story around this? Like can people even move from rllib 2+ to 3?

kouroshHakha · 2023-11-16T04:32:01Z

rllib/core/models/torch/encoder.py

@@ -285,30 +285,31 @@ def __init__(self, config: RecurrentEncoderConfig) -> None:
            bias=config.use_bias,
        )

+        self.state_in_out_spec = {


make this private attribute?

kouroshHakha · 2023-11-16T05:16:32Z

rllib/utils/spaces/space_utils.py

@@ -205,6 +205,56 @@ def flatten_to_single_ndarray(input_):
    return input_


+@DeveloperAPI
+def batch(list_of_structs, individual_items_already_have_batch_1: bool = False):


data types please (for input and output)

can we have unittest of this ?

done and done

also enhanced the docstring to make the example and explanations more clear.

kouroshHakha · 2023-11-16T05:21:32Z

rllib/utils/spaces/space_utils.py

+            flat = [[] for _ in range(len(flattened_item))]
+        for i, value in enumerate(flattened_item):
+            flat[i].append(value)
+


add:

if item is None: raise ValueError("Input list_of_structs does not contain valid structs.")

kouroshHakha · 2023-11-16T05:21:57Z

rllib/utils/spaces/space_utils.py

+        in this struct represents the batch for a single component
+        (in case struct is tuple/dict). Alternatively, a simple batch of
+        primitives (non tuple/dict) might be returned.
+    """


add

if not list_of_structs: raise ValueError("Input list_of_structs is empty.")

Signed-off-by: sven1977 <[email protected]>

sven1977 · 2023-11-16T11:16:36Z

Thanks for the review @kouroshHakha ! Waiting for tests to pass ...

Signed-off-by: sven1977 <[email protected]>

…anups

…and fixes. (ray-project#41074)

…41074) (#41212)

…rV2` API. (ray-project#41074) (ray-project#41212)

wip

017dcfc

Signed-off-by: sven1977 <[email protected]>

sven1977 requested review from avnishn, ArturNiederfahrenhorst, smorad, maxpumperla and kouroshHakha as code owners November 10, 2023 14:17

sven1977 assigned kouroshHakha Nov 10, 2023

sven1977 commented Nov 14, 2023

View reviewed changes

rllib/env/single_agent_env_runner.py Outdated

gym.register(

"custom-env-v0",

partial(

if (

Copy link

Contributor Author

sven1977 Nov 14, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug fix.

sven1977 commented Nov 14, 2023

View reviewed changes

sven1977 mentioned this pull request Nov 14, 2023

[RLlib] New ConnectorV2 API #02: SingleAgentEpisode enhancements. #41075

Merged

8 tasks

kouroshHakha approved these changes Nov 16, 2023

View reviewed changes

wip

768b88c

Signed-off-by: sven1977 <[email protected]>

sven1977 added 2 commits November 16, 2023 12:27

wip

1b7d1cc

Signed-off-by: sven1977 <[email protected]>

Merge branch 'master' into env_runner_support_connectors_01_minor_cle…

aed527f

…anups

sven1977 merged commit ca29fec into ray-project:master Nov 17, 2023
14 of 15 checks passed

sven1977 deleted the env_runner_support_connectors_01_minor_cleanups branch November 17, 2023 11:29

rickyyx mentioned this pull request Nov 22, 2023

[ci][core] Perf regression on tasks_per_second, pgs_per_second #41338

Closed

ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Nov 29, 2023

[RLlib] New ConnectorV2 API ray-project#1: Some preparatory cleanups …

f608186

…and fixes. (ray-project#41074)

sven1977 added a commit that referenced this pull request Dec 21, 2023

[RLlib] New ConnectorV2 API #3: Introduce actual ConnectorV2 API. (#…

bd555a0

…41074) (#41212)

vickytsang pushed a commit to ROCm/ray that referenced this pull request Jan 12, 2024

[RLlib] New ConnectorV2 API ray-project#3: Introduce actual `Connecto…

ad4e256

…rV2` API. (ray-project#41074) (ray-project#41212)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RLlib] Preparatory PR: Make EnvRunners use (enhanced) Connector API (#01: mostly cleanups and small fixes) #41074

[RLlib] Preparatory PR: Make EnvRunners use (enhanced) Connector API (#01: mostly cleanups and small fixes) #41074

sven1977 commented Nov 10, 2023 •

edited

Loading

sven1977 Nov 14, 2023

sven1977 Nov 14, 2023

kouroshHakha Nov 16, 2023

kouroshHakha Nov 16, 2023

sven1977 Nov 14, 2023

kouroshHakha Nov 16, 2023

sven1977 Nov 16, 2023

sven1977 Nov 14, 2023

sven1977 Nov 14, 2023

sven1977 Nov 14, 2023

sven1977 Nov 14, 2023

sven1977 Nov 14, 2023

kouroshHakha Nov 16, 2023

kouroshHakha Nov 16, 2023

kouroshHakha Nov 16, 2023

kouroshHakha Nov 16, 2023

kouroshHakha Nov 16, 2023

sven1977 Nov 16, 2023

sven1977 Nov 16, 2023

kouroshHakha Nov 16, 2023

sven1977 Nov 16, 2023

kouroshHakha Nov 16, 2023

sven1977 Nov 16, 2023

sven1977 commented Nov 16, 2023

		@@ -205,6 +205,56 @@ def flatten_to_single_ndarray(input_):
		return input_


		@DeveloperAPI

[RLlib] Preparatory PR: Make EnvRunners use (enhanced) Connector API (#01: mostly cleanups and small fixes) #41074

[RLlib] Preparatory PR: Make EnvRunners use (enhanced) Connector API (#01: mostly cleanups and small fixes) #41074

Conversation

sven1977 commented Nov 10, 2023 • edited Loading

Why are these changes needed?

Related issue number

Checks

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sven1977 commented Nov 16, 2023

sven1977 commented Nov 10, 2023 •

edited

Loading