In-Person Poster presentation / poster accept
Neural Groundplans: Persistent Neural Scene Representations from a Single Image
Prafull Sharma · Ayush Tewari · Yilun Du · Sergey Zakharov · Rares Ambrus · Adrien Gaidon · William Freeman · Fredo Durand · Joshua B Tenenbaum · Vincent Sitzmann
MH1-2-3-4 #145
Keywords: [ Scene Understanding ] [ nerf ] [ Neural scene representations ] [ 3d ] [ object-centric representations ] [ neural rendering ] [ Unsupervised and Self-supervised learning ]
We present a method to map 2D image observations of a scene to a persistent 3D scene representation, enabling novel view synthesis and disentangled representation of the movable and immovable components of the scene. Motivated by the bird’s-eye-view (BEV) representation commonly used in vision and robotics, we propose conditional neural groundplans, ground-aligned 2D feature grids, as persistent and memory-efficient scene representations. Our method is trained self-supervised from unlabeled multi-view observations using differentiable rendering, and learns to complete geometry and appearance of occluded regions. In addition, we show that we can leverage multi-view videos at training time to learn to separately reconstruct static and movable components of the scene from a single image at test time. The ability to separately reconstruct movable objects enables a variety of downstream tasks using simple heuristics, such as extraction of object-centric 3D representations, novel view synthesis, instance-level segmentation, 3D bounding box prediction, and scene editing. This highlights the value of neural groundplans as a backbone for efficient 3D scene understanding models.