Predicting Grokking Long Before it Happens: A look into the loss landscape of models which grok

This paper presents a cost-effective method for predicting grokking in neural networks—delayed perfect generalization following overfitting or memorization. By analyzing the learning curve of the first few epochs, we show that certain oscillations forecast grokking in extended training. Our approach, using the Fourier transform's \emph{spectral signature}, efficiently detects these oscillations. Additional experiments explore their origins and characterize the loss landscape.