Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters

Vision Language Models (VLMs) have demonstrated strong capabilities across various visual understanding and reasoning tasks, driven by incorporating image representations into the token inputs of Large Language Models (LLMs). However, their real-world deployment is often constrained by high latency during inference due to the substantial compute required by the LLM to process the large number of input tokens, predominantly arising from the image. To reduce inference costs, one can either downsize the LLM or reduce the number of input tokens needed to represent the image, the latter of which has been the focus of many recent efforts around token compression. However, it is unclear what the optimal trade-off is given a fixed inference budget. We characterize this optimal trade-off between the number of visual tokens and LLM parameters by establishing scaling laws that capture variations in performance with these two factors. Our results reveal a surprising trend: for visual reasoning tasks, the inference-optimal behavior heavily favors optimizing compute to utilize larger LLMs by reducing the visual token count - $\textit{even to a single token}$ in certain circumstances. While the token reduction literature has mainly focused on maintaining base model performance by modestly reducing the token count (e.g., $5\text{ - }10\times$), our results indicate that the compute-optimal inference regime requires operating under even higher token compression ratios. Our work underscores the performance and efficiency benefits of operating in lower visual token regimes compared to current token reduction literature and the importance of developing tailored token reduction algorithms for such conditions.