Denoising is not the End: Discrete Diffusion Language Models with Self-Correction

Discrete Diffusion Models have emerged as an effective approach to text generation, providing an alternative to autoregressive models. However, the generated texts often suffer from quality issues, including grammatical errors, lack of fluency and factual inaccuracies. This paper systematically evaluates different self-correction strategies for hybrid-noise (masking and uniform) discrete diffusion language models. The outputs of the base model are compared with self-corrected outputs under different strategies, with improvements measured in terms of quality, fluency, and fidelity. Our experiments show significant quality gains from iterative self-correction, with improvements in grammar, factuality, clarity, and creativity reaching double-digit percentages. We also observe a tradeoff between text quality and content preservation and identify optimal configurations that achieve significant quality enhancements while maintaining high fidelity to the original output. Finally, we find that it is optimal to allocate inference compute to denoising and self-correction in approximately equal proportions.