Skip to yearly menu bar Skip to main content


Replay across Experiments: A Natural Extension of Off-Policy RL

Dhruva Tirumala · Thomas Lampe · Jose Enrique Chen · Tuomas Haarnoja · Sandy Huang · Guy Lever · Ben Moran · Tim Hertweck · Leonard Hasenclever · Martin Riedmiller · Nicolas Heess · Markus Wulfmeier

Halle B #201
[ ]
Tue 7 May 7:30 a.m. PDT — 9:30 a.m. PDT


Replaying data is a principal mechanism underlying the stability and data efficiency of off-policy reinforcement learning (RL).We present an effective yet simple framework to extend the use of replays across multiple experiments, minimally adapting the RL workflow for sizeable improvements in controller performance and research iteration times.At its core, Replay across Experiments (RaE) involves reusing experience from previous experiments to improve exploration and bootstrap learning while reducing required changes to a minimum in comparison to prior work. We empirically show benefits across a number of RL algorithms and challenging control domains spanning both locomotion and manipulation, including hard exploration tasks from egocentric vision. Through comprehensive ablations, we demonstrate robustness to the quality and amount of data available and various hyperparameter choices. Finally, we discuss how our approach can be applied more broadly across research life cycles and can increase resilience by reloading data across random seeds or hyperparameter variations.

Chat is not available.