NeurIPS 2020 · 2020 · Spotlight
Learning Object-Centric Representations of Multi-object Scenes from Multiple Views
A single image often leaves an object’s shape or position ambiguous. The Multi-View and Multi-Object Network (MulMON) learns a shared object-centric representation from several observations of the same scene, using complementary viewpoints to build a more complete account of its objects.
Working example
A consistent scene across views
MulMON iteratively updates latent object representations as it receives more views, addressing the challenge of maintaining object correspondence across observations. During training, it must predict the appearance of the scene from novel viewpoints, encouraging the latent variables to capture spatial information beyond any single image.
Experiments demonstrate more accurate and disentangled object representations than the compared single-view methods. The model can also predict object segmentations at novel viewpoints, linking multi-view reconstruction with object-level scene understanding.