Expressing and Exploiting Parallelism in Language Model Decoding

For autoregressive language models, decoding naturally occurs sequentially, generating tokens one after another.Recent attempts to introduce parallelism require a pre-determined structure to implement parallel generation, such as generating an outline and dividing the responses into parallel sub-tasks.In this work we explore a new technique to automate parallel generation by dynamically exploiting various parallel structure in the semantics of the language model response.Specifically, we introduce a simple annotation language MSG that allows language models to express parallelism in their outputs.We then develop an interpreter for MSG that performs on-the-fly parallel generation during decoding, exploiting the parallelism expressed in the MSG-annotated outputs.We demonstrate that our approach can improve tokens generated per second by 21\% while maintaining the same quality of output.