GrabS: Generative Embodied Agent for 3D Object Segmentation without Scene Supervision

We study the hard problem of 3D object segmentation in complex point cloudswithout requiring human labels of 3D scenes for supervision. By relying on thesimilarity of pretrained 2D features or external signals such as motion to group 3Dpoints as objects, existing unsupervised methods are usually limited to identifyingsimple objects like cars or their segmented objects are often inferior due to thelack of objectness in pretrained features. In this paper, we propose a new two-stage pipeline called GrabS. The core concept of our method is to learn generativeand discriminative object-centric priors as a foundation from object datasets in thefirst stage, and then design an embodied agent to learn to discover multiple ob-jects by querying against the pretrained generative priors in the second stage. Weextensively evaluate our method on two real-world datasets and a newly createdsynthetic dataset, demonstrating remarkable segmentation performance, clearlysurpassing all existing unsupervised methods.