Bootstrapped Meta-Learning

Sebastian Flennerhag · Yannick Schroecker · Tom Zahavy · Hado van Hasselt · David Silver · Satinder Singh

Keywords: [ meta-gradients ] [ meta-reinforcement learning ] [ meta-learning ]

award Outstanding Paper
[ Abstract ]
[ Visit Poster at Spot C1 in Virtual World ] [ OpenReview
Tue 26 Apr 10:30 a.m. PDT — 12:30 p.m. PDT
Oral presentation: Oral 3: Meta-learning and adaptation
Wed 27 Apr 9 a.m. PDT — 10:30 a.m. PDT


Meta-learning empowers artificial intelligence to increase its efficiency by learning how to learn. Unlocking this potential involves overcoming a challenging meta-optimisation problem. We propose an algorithm that tackles this problem by letting the meta-learner teach itself. The algorithm first bootstraps a target from the meta-learner, then optimises the meta-learner by minimising the distance to that target under a chosen (pseudo-)metric. Focusing on meta-learning with gradients, we establish conditions that guarantee performance improvements and show that metric can be used to control meta-optimisation. Meanwhile, the bootstrapping mechanism can extend the effective meta-learning horizon without requiring backpropagation through all updates. We achieve a new state-of-the art for model-free agents on the Atari ALE benchmark and demonstrate that it yields both performance and efficiency gains in multi-task meta-learning. Finally, we explore how bootstrapping opens up new possibilities and find that it can meta-learn efficient exploration in an epsilon-greedy Q-learning agent - without backpropagating through the update rule.

Chat is not available.