Poster Sat, Apr 25, 2026 • 6:30 AM – 9:00 AM PDT Pavilion 4 P4-#4004

Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation

Sayash Kapoor ⋅ Benedikt Stroebl ⋅ Peter Kirgis ⋅ Nitya Nadgir ⋅ Zachary Siegel ⋅ Boyi Wei ⋅ Tianci Xue ⋅ Ziru Chen ⋅ Felix Chen ⋅ Saiteja Utpala ⋅ Franck Ndzomga ⋅ Dheeraj Oruganty ⋅ Sophie Luskin ⋅ Kangheng Liu ⋅ Botao Yu ⋅ Amit Arora ⋅ Dongyoon Hahm ⋅ Harsh Trivedi ⋅ Huan Sun ⋅ Juyong Lee ⋅ Tengjun Jin ⋅ Yifan Mai ⋅ Yifei Zhou ⋅ Yuxuan Zhu ⋅ Rishi Bommasani ⋅ Daniel Kang ⋅ Dawn Song ⋅ Peter Henderson ⋅ Yu Su ⋅ Percy Liang ⋅ Arvind Narayanan

Abstract

AI agents have been developed for complex real-world tasks from coding to customer service. But AI agent evaluations suffer from many challenges that undermine our understanding of how well agents really work (Figure 1). We introduce the Holistic Agent Leaderboard (HAL) to address these challenges. We make three main contributions. First, we provide a standardized evaluation harness that orchestrates parallel evaluations across hundreds of VMs, reducing evaluation time from weeks to hours while eliminating common implementation bugs. Second, we conduct three-dimensional analysis spanning models, scaffolds, and benchmarks. We validate the harness by conducting 21,730 agent rollouts across 9 models and 9 benchmarks in coding, web navigation, science, and customer service with a total cost of about $40,000. Our analysis reveals surprising insights, such as higher reasoning effort reducing accuracy in the majority of runs. Third, we use LLM-aided log inspection to uncover previously unreported behaviors, such as searching for the benchmark on HuggingFace instead of solving a task, or misusing credit cards in flight booking tasks. We share all agent logs, comprising 2.5B tokens of language model calls, to incentivize further research into agent behavior. By standardizing how the field evaluates agents and addressing common pitfalls in agent evaluation, we hope to shift the focus from agents that ace benchmarks to agents that work reliably in the real world.