A Cognitive Lens on Selective Memory in Neural Sequence Models: Surprise, Replay, and Consolidation

Recent sub-quadratic sequence models-including Mamba, DeltaNet, and Titans-have independently converged on selective memory mechanisms allocating write intensity proportional to input surprise, mirroring surprise-gated encoding in human episodic memory. We argue this convergence reflects shared computational pressures studied under complementary learning systems (CLS) theory. However, current models capture only encoding; the critical post-encoding processes of consolidation, replay, and reconsolidation remain absent. Drawing triangular connections among (i) prioritized experience replay in reinforcement learning, (ii) surprise-gated writing in sequence models, and (iii) sleep-dependent consolidation in cognition, we identify three architectural gaps: lack of offline replay across memory systems, absence of importance-sampling correction for selective writes, and missing reconsolidation pathways. We formalize each gap, propose mechanisms, and derive testable predictions at both machine learning and cognitive modeling levels, charting a design space for cognitively grounded sequence architectures.