UGG: Unified Generative Geometry Model

Condition on partial observations. Complete the scene in RGB, depth, and camera geometry.

Abstract

Understanding 4D scenes is a core challenge in visual intelligence, requiring models to jointly capture scene geometry, appearance, and dynamics from partial observations. Existing approaches, however, are subdivided into feedforward models that are used for direct geometric prediction and generative models for visual synthesis. We propose a unified generative-predictive framework for 4D scene understanding that jointly models RGB video and geometric representations such as depth and camera pose with a single joint generative model. Our generative formulation enables us to learn from diverse, heterogeneous datasets with different subsets of geometric and visual annotations, and enables us to condition on arbitrary subsets of inputs, allowing a single model to perform cross-modal generation, geometric prediction, and future scene inference from partial observations. In addition, our generative approach enables test-time search over multiple candidate scene completions, improving geometric prediction from sparse observations. Overall, we illustrate the efficacy of our generative-predictive framework on a suite of generation and prediction tasks.

Results

Choose a task. See what goes in, and what UGG generates.

3D Reconstruction NVS Depth Prediction Camera Prediction Text-to-4D

3D Reconstruction

Explore reconstructed scene geometry in an interactive 3D view.

3D reconstruction · placeholder

Explore the scene in 3D

The selected reconstruction and interactive viewer will go here.

Choose a case

Novel View Synthesis

Generate unseen views given observed frames and camera geometry.

Loading the provisional novel-view example…

Choose a case

Depth Prediction

Infer scene geometry from observed RGB frames.

Loading the provisional depth example…

Choose a case

Camera Prediction

Recover camera geometry from visual observations.

Loading the provisional camera example…

Choose a case

Text-to-4D Generation

Generate appearance, depth, and camera motion from a text prompt.

Loading the generated scene…

Choose a case

Method

Generation and prediction through a shared model of the scene.

A shared transformer models appearance and geometry across modalities.

A shared scene representation

UGG represents RGB, depth, and camera geometry as spatially aligned modalities. A multimodal transformer exchanges information through shared self-attention and uses modality-specific layers.

Condition on what is observed

Clean tokens provide context, while noisy tokens mark components to generate. Varying this pattern across modality and time lets the same model answer different prediction and generation queries.

Search over scene hypotheses

The generative formulation supports multiple candidate scene completions. The paper explores selecting candidates using cross-modal consistency measured through photometric reprojection.

Benchmarks

One model for generation and geometric prediction. Explore the results by task, observation setting, and dataset.

See Tables 1–4 in the paper. Interactive charts load when JavaScript is available.

BibTeX

Cite UGG.

@inproceedings{kiruga2026ugg,
  title={UGG: Unified Generative Geometry Model},
  author={Kiruga, Amani and Li, Zhiyi and Cai, Ruojin and
          Lillemark, Hansen and Wu, Xiaoshan and Tewari, Ayush and Du, Yilun},
  year={2026},
  booktitle={Advances in Neural Information Processing Systems}
}

Publication details will be updated with the final paper link.