Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaboration

We present Collaborative Gym (Co-Gym), a general framework for developing and evaluating collaborative agents that engage in asynchronous, bidirectional communication with humans while interacting with task environments. While fully autonomous agents typically operate without humans in the loop, numerous use cases inherently require agents to collaborate with humans due to humans' latent preferences, domain expertise, or the need for control. This motivates the study of collaborative agents designed to work together with humans. We instantiate Co-Gym with three representative tasks---creating travel plans, writing Related Work sections, and analyzing tabular data---in both simulated and real-world conditions, and propose an evaluation framework assessing both collaboration outcomes and processes. Auditing initiative-taking patterns reveals that simply expanding the agent's action space to include communication is insufficient for effective collaboration or appropriate initiative. By equipping agents with a situational planning module, we observe substantial gains: the best-performing collaborative agents consistently outperform their fully autonomous counterparts in task performance, achieving win rates of 86% in Travel Planning, 74% in Tabular Analysis, and 66% in Related Work when evaluated by real users. Despite these improvements, our evaluation reveals persistent limitations in current language models and agents, with communication and situational awareness failures observed in 65% and 80% of cases in the real condition, respectively.