Distilled Feedforward Networks Are As Robust as Energy-Based Models trained with Equilibrium Propagation

Deep neural networks (DNNs) are not naturally robust to adversarial attacks on their inputs, leading to loss of reliability in a general use case. One of the state-of-the-art defenses against adversarial attacks is adversarial training, which introduces adversarial examples into the training set. While adversarially trained models are more robust to attacks, their accuracy on clean images drops and the additional robustness gained does not generalize well to different types of attacks. Previous studies have proposed energy-based models (EBMs) with a Hopfield-like energy function are inherently robust to adversarial perturbations without any drop in clean accuracy. However, EBMs trained with equilibrium propagation require attaining a fixed point during their dynamical evolution, thus making inference a time consuming process on traditional digital hardware as opposed to neuromorphic hardware which is well-suited for such minimization problems. In this work we report that by training feedforward networks to mimic the fixed points of EBMs, we achieve similar robustness but at drastically shorter inference times. We demonstrate the adversarial robustness conferred by EBM distillation using both white-box and black-box attacks as well as natural corruptions on the CIFAR-10 and CIFAR-100 dataset. We thus posit that EBM distillation could provide an alternative method to adversarial training.