ReplayBuffer size with on-policy algorithm

[jvasso]

Hi,
Can someone confirm that when using an on-policy algorithm like PPO, the size of the ReplayBuffer should be equal to the value of step_per_collect? Unless I'm mistaken, on-policy algorithms require the data used for training to be collected from the same policy that is currently being learned. Therefore, if the ReplayBuffer size exceeds step_per_collect, the next policy update might be based on data collected with old versions of the policy, which could distort the gradient.

[MischaPanch]

@jvasso essentially yes, it should not be smaller. If it's larger that's not a problem, since the buffer will be reset before the next learning step.

This structure (of having to deal with the buffer size at all for an on-policy algo) is one of Tianshou's main technical debts, and we're planning to address this with a major refactoring that will be part of Tianshou 2.0.0