Understanding Scaling Laws With Token-Level Analysis

Large language models (LLMs) exhibit remarkably smooth power-law scaling of cross-entropy loss with respect to training data size (tokens). Despite the robustness of this empirical law, its mechanistic origins remain unclear: why do the dynamics of gradient-based training yield such a clean functional form? This paper attacks the question through a token-level lens. We pretrain OLMo-2 across multiple token budgets and log per-token validation losses across checkpoints. We find (i) individual token-loss trajectories are highly heterogeneous and often noisy, with no obvious shared parametric form; (ii) nevertheless, best-fit power laws typically imply decreasing trends, and a substantial fraction of tokens are well-approximated by power laws; and (iii) the canonical aggregate power law emerges only after averaging over a critical mass of token-level losses, with fit error decaying rapidly as coverage increases. Our findings show that power-law scaling is not a universal property of each token's learning curve, but an emergent phenomenon arising from averaging many heterogeneous token-wise dynamics.