Synthetic Error Injection Fails to Elicit Self-Correction In Language Models

Reasoning language models trained with reinforcement learning have been observed to exhibit emergent self-correction behavior. Inspired by the success of synthetic error injection in autonomous driving and robotics, we attempt to directly teach models to self-correct without reinforcement learning. Our approach inserts artificial errors into reasoning chains, masks them, and supervises the model to recognize and correct these mistakes. Despite the intuitive appeal of this method, we find that it fails to significantly improve performance even on simple synthetic tasks across multiple models. Moreover, even when the model catches its own error, it often parrots the original mistake. We find that the distribution shift of synthetic errors to on-policy errors significantly degrades the error-correction capabilities of the fine-tuned model, even with good synthetic coverage of on-policy errors. Our results help explain why on-policy reinforcement learning methods have proven uniquely effective for eliciting self-correction.