The meaning of some settings in the SafeLOOP algorithm.

[TomatoZ2]

I have read the paper Learning Off-Policy with Online Planning and took a quick look at the code of SafeLOOP, but I'm not clear about the meaning and function of some algorithm configs. Could you please explain the configs below?

dynamics_cfgs
elite_size, use_decay
planner_cfgs
num_iterations, num_particles, num_samples, temperature, cost_temperature
algo_cfgs
action_repeat(I'm very confused about what the function this config is, why action have to be repeated several times?)

[Gaiejj]

Certainly! Here's the explanation for these parameters:

1. elite_size: This signifies the number of elite individuals selected from sampled individuals. It is commonly used in action planning of a model-based RL. (More details can refer to Constrained Model-based Reinforcement Learning with Robust Cross-Entropy Method and The Cross-Entropy Method for Combinatorial and Continuous Optimization)

2. use_decay: This parameter indicates whether to use the weight decay technique when updating the dynamic model.

3. num_iterations: This is the number of iterations performed in planning.

4. num_particles: During planning, this is the number of models used for evaluating an action trajectory.

5. num_samples: This indicates the number of trajectories sampled.

6. temperature: This is the weight for updating the action mean. It can control the extent of exploration and exploitation during learning.

[TomatoZ2]

Thanks for your reply, and what about the action_repeat, why action have to be repeated several times?

[Gaiejj]

Hello, action_repeat entails repeating an action several times, conserving time, and accelerating training.

[TomatoZ2]

Got it, this means that this parameter is only suitable for environments with high frequency of action, and for general environments, just set this parameter to 1, right?

[Gaiejj]

Sure, feel free to set that to 1 according to your environment.

[TomatoZ2]

Does SafeLOOP work in a vector environment? Is it feasible to directly modify the env in the adapter to a vector environment?

[TomatoZ2]

When the safeloop algorithm is applied in a custom environment, the algorithm cannot converge. So I decided to invalidate the cost limit until a certain epoch and then reinstate the cost limit. When the limits are invalidated, the agent can learn effectively and gradually improve its performance. However, as soon as the restriction is reinstated, SafeArc's actions immediately become very bad. Note that the cost of most trajectories still meet the limit at this time, safeARC still utilize the reward to update. Why is this happening? How do I fix the problem?

[TomatoZ2]

I suspect there is a bug in this section of the SafeARC code, where mean_episode_returns is a two-dimensional tensor, and .nonzero().reshape(-1) will cause elite_values to contain a lot of irrelevant data (index=0).

if feasible_num < self._num_elites:
    elite_values, elite_actions = -mean_episode_costs, actions
else:
    elite_idxs = (
        (mean_episode_costs <= self._cost_limit).nonzero().reshape(-1)
    )  # like tensor([0, 1])
    elite_values, elite_actions = mean_episode_returns[elite_idxs], actions[:, elite_idxs]

example as follow

import torch

a = torch.rand(10, 1)
elite_idxs = (a <= 0.5).nonzero().reshape(-1)
print(a[elite_idxs])

tensor([[0.2820],
        [0.7670],
        [0.1507],
        [0.7670],
        [0.2072],
        [0.7670],
        [0.3530],
        [0.7670],
        [0.0151],
        [0.7670]])

[Gaiejj]

We are currently reviewing the relevant code and will provide feedback as soon as possible.

[hdadong]

Thank you for highlighting this critical issue. Following our verification, we confirm it as a performance-impacting bug. We plan to address this in an upcoming update. Your alert is greatly appreciated.

[TomatoZ2]

When I use the safeloop algorithm, I have to limit the output of the environment model due to the specificity of the environment. But in some cases, arc will produce harmful actions when the real environment should terminate because the output of the environment model reaches the boundary at this time, and the state generated by any action at this time is the same. How do I deal with the termination of my environment model? How to avoid arc's harmful actions and train the agent in this case?

[hdadong]

Based on your description, it appears that in certain scenarios, your environment may reach a termination state. To address this, you should implement a termination_function. This function is critical during the planning phase, as it evaluates each state to determine whether it is a terminal state. If a state is identified as terminal, the reward of the trajectory containing this state should be set to zero. This adjustment ensures that the planning algorithm does not select actions corresponding to these trajectories. For a practical implementation, you might find the following references:
The implementation of a termination function in ARC can be examined at ARC Controller Implementation.
An example of how this is integrated into an environment is available at Ant Environment Implementation.
I hope this information assists you in your work.