Overcoming The Spectral Bias of Neural Value Approximation

Ge Yang · Anurag Ajay · Pulkit Agrawal

Keywords: [ reinforcement learning ] [ neural tangent kernels ] [ q learning ]

[ Abstract ]
[ Visit Poster at Spot H0 in Virtual World ] [ OpenReview
Thu 28 Apr 10:30 a.m. PDT — 12:30 p.m. PDT


Value approximation using deep neural networks is at the heart of off-policy deep reinforcement learning, and is often the primary module that provides learning signals to the rest of the algorithm. While multi-layer perceptrons are universal function approximators, recent works in neural kernel regression suggest the presence of a \textit{spectral bias}, where fitting high-frequency components of the value function requires exponentially more gradient update steps than the low-frequency ones. In this work, we re-examine off-policy reinforcement learning through the lens of kernel regression and propose to overcome such bias via a composite neural tangent kernel. With just a single line-change, our approach, the Fourier feature networks (FFN) produce state-of-the-art performance on challenging continuous control domains with only a fraction of the compute. Faster convergence and better off-policy stability also make it possible to remove the target network without suffering catastrophic divergences, which further reduces (\text{TD}(0))'s bias to over-estimate the value.

Chat is not available.