Symmetry, Gauss-Newton, and Whitening in Neural Network Optimization

Neural networks have architecture dependent parameter space symmetries where distinct parameters realize the same function. If an optimizer does not behave consistently under symmetry transformations, optimization dynamics depend on the arbitrary choice of representative along symmetry orbits. Recent work characterizes symmetries via the Jacobian of the evaluation map, decomposing parameter space into a \emph{functional subspace} affecting outputs and a \emph{fiber} subspace of symmetry directions. We show Gauss-Newton (GN) is symmetry-equivariant and that symmetry equivariance forces the pullback form $P = J^\top B J$ within a natural class of preconditioners. We then show that GN uniquely achieves optimal convergence conditioning within this class, while whitening achieves optimal whitening conditioning (isotropy of updates) but exhibits structured symmetry breaking due to the square root. These results clarify the relationship between Gauss-Newton, whitening and symmetries, which could be especially valuable for overparameterized networks.