In-Person Poster presentation / top 25% paper

Generating Diverse Cooperative Agents by Learning Incompatible Policies

Rujikorn Charakorn · Poramate Manoonpong · Nat Dilokthanakul

MH1-2-3-4 #118

Keywords: [ Reinforcement Learning ] [ reinforcement learning ] [ diversity ] [ robustness ] [ Collaboration ] [ cooperation ] [ multi-agent systems ]

[ Abstract ] [ Project Page ]
[ Slides [ Poster [ OpenReview
Wed 3 May 2:30 a.m. PDT — 4:30 a.m. PDT
Oral presentation: Oral 5 Track 5: Deep Learning and representational learning & Reinforcement Learning
Wed 3 May 1 a.m. PDT — 2:30 a.m. PDT


Training a robust cooperative agent requires diverse partner agents. However, obtaining those agents is difficult. Previous works aim to learn diverse behaviors by changing the state-action distribution of agents. But, without information about the task's goal, the diversified agents are not guided to find other important, albeit sub-optimal, solutions: the agents might learn only variations of the same solution. In this work, we propose to learn diverse behaviors via policy compatibility. Conceptually, policy compatibility measures whether policies of interest can coordinate effectively. We theoretically show that incompatible policies are not similar. Thus, policy compatibility—which has been used exclusively as a measure of robustness—can be used as a proxy for learning diverse behaviors. Then, we incorporate the proposed objective into a population-based training scheme to allow concurrent training of multiple agents. Additionally, we use state-action information to induce local variations of each policy. Empirically, the proposed method consistently discovers more solutions than baseline methods across various multi-goal cooperative environments. Finally, in multi-recipe Overcooked, we show that our method produces populations of behaviorally diverse agents, which enables generalist agents trained with such a population to be more robust.See our project page at

Chat is not available.