Persistent internal state helps maintain learning plasticity

A failure mode associated with continual learning is a loss of plasticity, in that a neural network that is repeatedly trained eventually ceases to be able to adapt to changing stimuli. In this work we show that plasticity loss can be mitigated by maintaining a new internal state -- persistent from batch to batch -- in the network. In training, the weights of the network are updated by conventional gradient descent, and this extra internal state is updated by a learned rule. This internal state provides an extra mechanism for the network to retain a memory of previous problems, in addition to the conventional weights. We empirically demonstrate the benefits in simple benchmarks.