GradMax: Growing Neural Networks using Gradient Information

Utku Evci · Bart van Merrienboer · Thomas Unterthiner · Fabian Pedregosa · Max Vladymyrov

Keywords: [ computer vision ] [ architecture search ] [ Efficient training ] [ efficient ]

[ Abstract ]
[ Visit Poster at Spot J1 in Virtual World ] [ OpenReview
Tue 26 Apr 10:30 a.m. PDT — 12:30 p.m. PDT


The architecture and the parameters of neural networks are often optimized independently, which requires costly retraining of the parameters whenever the architecture is modified. In this work we instead focus on growing the architecture without requiring costly retraining. We present a method that adds new neurons during training without impacting what is already learned, while improving the training dynamics. We do this by maximizing the gradients of the new neurons and find an approximation to the optimal initialization by means of the singular value decomposition (SVD). We call this technique Gradient Maximizing Growth (GradMax) and demonstrate its effectiveness in variety of vision tasks and architectures.

Chat is not available.