Skip to yearly menu bar Skip to main content


Poster

EMOS: Embodiment-aware Heterogeneous Multi-robot Operating System with LLM Agents

Junting Chen · Checheng Yu · Xunzhe Zhou · Tianqi Xu · Yao Mu · Mengkang Hu · Wenqi Shao · Yikai Wang · Guohao Li · Lin Shao

Hall 3 + Hall 2B #30
[ ] [ Project Page ]
Thu 24 Apr midnight PDT — 2:30 a.m. PDT

Abstract:

Heterogeneous multi-robot systems (HMRS) have emerged as a powerful ap-proach for tackling complex tasks that single robots cannot manage alone. Currentlarge-language-model-based multi-agent systems (LLM-based MAS) have shownsuccess in areas like software development and operating systems, but applyingthese systems to robot control presents unique challenges. In particular, the ca-pabilities of each agent in a multi-robot system are inherently tied to the physicalcomposition of the robots, rather than predefined roles. To address this issue,we introduce a novel multi-agent framework designed to enable effective collab-oration among heterogeneous robots with varying embodiments and capabilities,along with a new benchmark named Habitat-MAS. One of our key designs isRobot Resume: Instead of adopting human-designed role play, we propose a self-prompted approach, where agents comprehend robot URDF files and call robotkinematics tools to generate descriptions of their physics capabilities to guidetheir behavior in task planning and action execution. The Habitat-MAS bench-mark is designed to assess how a multi-agent framework handles tasks that requireembodiment-aware reasoning, which includes 1) manipulation, 2) perception, 3)navigation, and 4) comprehensive multi-floor object rearrangement. The experi-mental results indicate that the robot’s resume and the hierarchical design of ourmulti-agent system are essential for the effective operation of the heterogeneousmulti-robot system within this intricate problem context.

Live content is unavailable. Log in and register to view live content