Skip to yearly menu bar Skip to main content


Virtual presentation / poster accept

Noise Injection Node Regularization for Robust Learning

Noam Levi · Itay Bloch · Marat Freytsis · Tomer Volansky

Keywords: [ deep learning ] [ Input corruption ] [ Reguralization ] [ Noise Injection ] [ Deep Learning and representational learning ]


Abstract: We introduce Noise Injection Node Regularization (NINR), a method of injecting structured noise into Deep Neural Networks (DNN) during the training stage, resulting in an emergent regularizing effect. We present theoretical and empirical evidence for substantial improvement in robustness against various test data perturbations for feed-forward DNNs when trained under NINR. The novelty in our approach comes from the interplay of adaptive noise injection and initialization conditions such that noise is the dominant driver of dynamics at the start of training. As it simply requires the addition of external nodes without altering the existing network structure or optimization algorithms, this method can be easily incorporated into many standard problem specifications. We find improved stability against a number of data perturbations, including domain shifts, with the most dramatic improvement obtained for unstructured noise, where our technique outperforms other existing methods such as dropout or $L_2$ regularization, in some cases. We further show that desirable generalization properties on clean data are generally maintained.

Chat is not available.