MUX: Continuous Reasoning via Multiplexed Tokens

Language models solve complex problems by articulating intermediate reasoning steps in natural language. While effective, this process is computationally bottlenecked: each reasoning step conveys only a single subword, and many steps are spent expressing a thought rather than carrying out computation. We propose MUX, a simple method for high-bandwidth and compact reasoning based on distillation of discrete reasoning into continuous multiplexed tokens. Each continuous token is trained to represent a weighted linear superposition (multiplexing) of a span of discrete reasoning steps, while ensuring that this superposition is lossless and the span can be fully recovered (demultiplexing). We prove that simple position-dependent weightings, such as properly chosen geometric decay, support lossless multiplexing, and further prove that multiplexed reasoning can perform parallel exploration in problems that require search. Across 16 evaluation settings spanning two language models, MUX is competitive with or outperforms strong continuous reasoning baselines, especially when the base discrete reasoning is verbose. Overall, our results suggest that appropriately chosen learning targets can make continuous reasoning both efficient and interpretable.