Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: evaluate_policy called multiple times vor vectorized environments #1912

Open
5 tasks done
LukasFehring opened this issue Apr 26, 2024 · 5 comments
Open
5 tasks done
Labels
documentation Improvements or additions to documentation help wanted Help from contributors is welcomed

Comments

@LukasFehring
Copy link

LukasFehring commented Apr 26, 2024

🐛 Bug

When calling

from stable_baselines3.common.evaluation import evaluate_policy
def custom_callback(locals, globals):
    pass

evaluate_policy(callback=custom_callback)

with a vecenv, then the callback gets executed for each of the environments separately. However, the locals dict contains the aggregated results. Therefore you have to manually check for which environment the callback was called, or only execute it every n_envs time.

To Reproduce

import gym
from stable_baselines3 import PPO
from stable_baselines3.common.vec_env import DummyVecEnv
from stable_baselines3.common.evaluation import evaluate_policy

# Define a simple callback function
def callback(_locals, _globals):
    pass

# Function to create multiple environments
def make_env():
    return gym.make('CartPole-v1')

# Number of environments
num_envs = 4
envs = [make_env for _ in range(num_envs)]

# Create vectorized environment
vec_env = DummyVecEnv(envs)

# Create a model
model = PPO("MlpPolicy", vec_env, verbose=1)

# Train the model
model.learn(total_timesteps=5000)

# Evaluate the policy
mean_reward, std_reward = evaluate_policy(model, vec_env, n_eval_episodes=10, callback=callback)

print("Mean reward:", mean_reward, "STD reward:", std_reward)

Relevant log output / Error message

No response

System Info

  • OS: Linux-3.10.0-1160.2.1.el7.x86_64-x86_64-with-glibc2.17 # 1 SMP Tue Oct 20 15:39:03 UTC 2020
  • Python: 3.8.19
  • Stable-Baselines3: 2.4.0a0
  • PyTorch: 2.3.0+cu121
  • GPU Enabled: False
  • Numpy: 1.24.4
  • Cloudpickle: 3.0.0
  • Gymnasium: 0.29.1
  • OpenAI Gym: 0.23.0

Checklist

  • My issue does not relate to a custom gym environment. (Use the custom gym env template instead)
  • I have checked that there is no similar issue in the repo
  • I have read the documentation
  • I have provided a minimal and working example to reproduce the bug
  • I've used the markdown code blocks for both code and stack traces.
@LukasFehring LukasFehring added the bug Something isn't working label Apr 26, 2024
@araffin
Copy link
Member

araffin commented Apr 26, 2024

Hello,
what is your usecase/expected behavior?

the for loop also decompose the info per env:

# unpack values so that the callback can access the local variables
reward = rewards[i]
done = dones[i]
info = infos[i]
episode_starts[i] = done
if callback is not None:
callback(locals(), globals())

@LukasFehring
Copy link
Author

How so? Both the globals and locals contain information on every environment in the vectorized environment. How am I supposed to determine for which env the callback is called?

@araffin
Copy link
Member

araffin commented May 7, 2024

there is the local variable "i"

@LukasFehring
Copy link
Author

Ah ok sorry then.
A documentation of locals and globals would probably help to find that! :)

@araffin araffin added documentation Improvements or additions to documentation help wanted Help from contributors is welcomed and removed bug Something isn't working labels May 7, 2024
@araffin
Copy link
Member

araffin commented May 10, 2024

A documentation of locals and globals would probably help to find that! :)

feel free to open a PR that updates the doc ;)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation help wanted Help from contributors is welcomed
Projects
None yet
Development

No branches or pull requests

2 participants