On the Cone Effect in the Learning Dynamics

Understanding the learning dynamics of neural networks is a central topic in deep learning community. In this paper, we take an empirical perspective to study the learning dynamics of neural networks in practical, real-world settings. Our key findings reveal a two-phase learning process: i) Phase I, characterized by highly non-linear dynamics indicative of the rich regime, and ii) Phase II, where dynamics continue to evolve but are constrained within a narrow space, a phenomenon we term the cone effect. This two-phase framework builds on the hypothesis proposed by \citet{fort2020deep}, but we uniquely identify and analyze the cone effect in Phase II, demonstrating its significant performance advantages over fully linearized training.