LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression

In long context scenarios, large language models (LLMs) face three main challenges: higher computational cost, performance reduction, and position bias. Research indicates that LLM performance hinges on the density and position of key information in the input prompt. Addressing this, we introduce LongLLMLingua, a method for prompt compression that improves LLMs’ key information perception, effectively tackling these challenges. Our extensive evaluation across various long context scenarios demonstrates that LongLLMLingua not only enhances performance but also significantly reduces costs and latency. For instance, in the NaturalQuestions benchmark, LongLLMLingua boosts performance by up to 21.4% with around 4x fewer tokens in GPT-3.5-Turbo, leading to substantial cost savings. It achieves a 94.2% cost reduction in the LooGLE benchmark. Moreover, when compressing prompts of about 10k tokens at rates of 2x-10x, LongLLMLingua can accelerate end-to-end latency by 1.4x-3.8x.