Divine Benevolence is an $x^2$: GLUs have asymptotically faster scaling laws than MLPs

Scaling laws can be understood from ground-up numerical analysis, where traditional function approximation theory can explain shifts in model architecture choices. GLU variants now dominate frontier LLMs and similar outer-product architectures are prevalent in ranking models. The success of these architectures has mostly been left as an empirical discovery. In this paper, we apply the tools of numerical analysis to expose a key factor: these models have an $x^2$ which enables asymptotically faster scaling than MLPs. Our key contribution is to demonstrate that the $L(P)$ scaling slope is $L(P ) \propto P^{−3}$ for GLUs but only $L(P ) = P^{−2}$ for MLPs on function reconstruction problems. We provide a parameter construction and empirical verification of these slopes for 1D function approximation. This opens the possibility of architecture design from first principles numerical theory to unlock superior scaling in large models.