This video introduces LiteReality, an automated pipeline designed to convert RGB-D scans of indoor environments into "graphics-ready" 3D scenes. The system delivers fully structured, realistic environments with object-level decomposition, physically based rendering (PBR) materials, and support for articulated objects and physics-based interactions. The pipeline operates in four key stages: scene parsing (detecting objects, room layout, and building a scene graph), object reconstruction (matching to a 3D database), material painting (transferring visual details and creating PBR materials), and procedural reconstruction (rebuilding the scene and enabling interactivity). LiteReality demonstrates robust performance across challenging conditions and is presented as a valuable tool for applications in robotics, virtual reality, and digital twin creation.
Summarizing LiteReality: Graphics-Ready 3D Scene Reconstruction from RGB-D Scans
0:02 Automatic Graphics-Ready Pipeline: LiteReality is presented as an automatic pipeline that transforms RGB-D scans into fully structured, realistic 3D environments ready for standard rendering pipelines.
0:37 Photorealistic & Structured Output: Reconstructed scenes feature object-level decomposition and are assigned physically based rendering (PBR) materials, achieving a near-photorealistic appearance.
0:54 Interactive Environments: The scenes generated by LiteReality are fully interactive, including articulated objects (e.g., doors, drawers) and support for physics-based interactions.
1:05 Flexible Relighting: The pipeline also enables flexible relighting capabilities within the reconstructed 3D environments.
1:10 Four Key Stages of Reconstruction: The LiteReality pipeline is broken down into four primary stages:
1:14 Scene Parsing: Involves detecting 3D bounding boxes, estimating room layout using pre-trained models, and building a scene graph to organize detections into a structured, physically plausible representation.
1:35 Object Reconstruction: Identical objects are grouped and matched to their closest high-quality 3D counterparts from a curated database.
1:46 Material Painting: Transfers visual details from the scan using reference images, applies an automatic crop mapping scheme, and uses language/vision-based retrieval for initial PBR material assignment, followed by albedo-only optimization.
2:20 Procedural Reconstruction: Rebuilds the scene step-by-step, starting with the room layout, windows, and doors, then furniture, and finally enabling interactive settings like object articulation.
2:32 Demonstrated Robustness: LiteReality's robustness is demonstrated on public datasets like ScanNet and other real-world RGB-D scans.
3:05 Superior Performance: Compared to other baselines, LiteReality achieves better visual similarity and remains robust under challenging conditions such as poor lighting, severe geometry misalignment, and diverse image styles.
3:30 Diverse Downstream Applications: The reconstructed scenes are readily usable for applications like relighting, physically based interaction, robotics, virtual reality (VR), and digital twin environments.