Skip to yearly menu bar Skip to main content


Poster
in
Workshop: Deep Generative Model in Machine Learning: Theory, Principle and Efficacy

On the Cone Effect in the Learning Dynamics

Zhanpeng Zhou · Yongyi Yang · Jie Ren · Mahito Sugiyama · Junchi Yan

Keywords: [ Learning Dynamics ] [ Cone Effect ]


Abstract:

Understanding the learning dynamics of neural networks is a central topic in deep learning community. In this paper, we take an empirical perspective to study the learning dynamics of neural networks in practical, real-world settings. Our key findings reveal a two-phase learning process: i) Phase I, characterized by highly non-linear dynamics indicative of the rich regime, and ii) Phase II, where dynamics continue to evolve but are constrained within a narrow space, a phenomenon we term the cone effect. This two-phase framework builds on the hypothesis proposed by \citet{fort2020deep}, but we uniquely identify and analyze the cone effect in Phase II, demonstrating its significant performance advantages over fully linearized training.

Chat is not available.