Stateful Active Facilitator: Coordination and Environmental Heterogeneity in Cooperative Multi-Agent Reinforcement Learning

In cooperative multi-agent reinforcement learning, a team of agents works togetherto achieve a common goal. Different environments or tasks may require varyingdegrees of coordination among agents in order to achieve the goal in an optimalway. The nature of coordination will depend on properties of the environment—itsspatial layout, distribution of obstacles, dynamics, etc. We term this variationof properties within an environment as heterogeneity. Existing literature has notsufficiently addressed the fact that different environments may have different levelsof heterogeneity. We formalize the notions of coordination level and heterogeneitylevel of an environment and present HECOGrid, a suite of multi-agent RLenvironments that facilitates empirical evaluation of different MARL approachesacross different levels of coordination and environmental heterogeneity by providinga quantitative control over coordination and heterogeneity levels of theenvironment. Further, we propose a Centralized Training Decentralized Executionlearning approach called Stateful Active Facilitator (SAF) that enables agents towork efficiently in high-coordination and high-heterogeneity environments througha differentiable and shared knowledge source used during training and dynamicselection from a shared pool of policies. We evaluate SAF and compare its performanceagainst baselines IPPO and MAPPO on HECOGrid. Our results showthat SAF consistently outperforms the baselines across different tasks and differentheterogeneity and coordination levels.