Shared Gradient Discovery and Superposition: Learning Dynamics of Generalization in LLMs
Andrei Mircea ⋅ Ildus Sadrtdinov ⋅ Irina Rish ⋅ Ekaterina Lobacheva
Abstract
We propose shared gradient discovery and superposition as a mechanism underlying generalization in LLMs, where shared gradients lead to inherently generalizing shared solutions. To validate our hypothesis, we study circuit emergence as one form of learning such generalizing solutions. We find that our hypothesis can indeed explain and shed new light on circuit emergence and generalization.
Chat is not available.
Successful Page Load