This video introduces "BecomingLit," a novel method for creating highly realistic, relightable 3D head avatars from multi-view light stage videos, capable of interactive rendering from new viewpoints. The project proposes a cost-effective custom light stage setup to capture a diverse dataset of subjects under varying illumination and expressions. At its core, BecomingLit employs a relightable avatar representation based on 3D Gaussian primitives animated by a parametric head model and an expression-dependent dynamics module. A key innovation is its hybrid neural shading approach, which combines a neural diffuse Bidirectional Reflectance Distribution Function (BRDF) with an analytical specular term, enabling disentangled material reconstruction and all-frequency relighting with both point lights and environment maps. The system also supports avatar animation and control from monocular videos. Extensive experiments demonstrate that BecomingLit significantly outperforms existing state-of-the-art methods in both relighting and reenactment tasks, producing photorealistic results even under challenging conditions.
BecomingLit: Relightable Gaussian Avatars with Hybrid Neural Shading
0:00 Introduction to BecomingLit: The video introduces "BecomingLit," a method for creating relightable Gaussian avatars with hybrid neural shading, aiming for high-quality, relightable head avatars from multi-view light stage videos.
0:10 Advanced Appearance Model: The method uses a learned neural BRDF formulation to model complex facial reflection behavior, enabling photorealistic renderings under novel and challenging illuminations.
0:33 Real-time & All-Frequency Relighting: The appearance model supports real-time rendering with both point lights and continuous illumination (e.g., environment maps), showcasing pore-level specular highlights and adaptive skin tone.
0:45 3D Gaussian Primitive Avatars: Avatars are constructed using 3D Gaussian primitives, animated by a parametric face model. The appearance model accurately recovers albedo, normals, and diffuse reflectance via a neural BRDF, combined with a specular BRDF for authentic rendering of skin, eyes, and hair.
1:14 Hybrid Neural Shading: A neural diffuse BRDF predicts reflectance values, while a second component predicts view-dependent normals and specular intensity for a Cook-Torrance BRDF, computing the specular part of the appearance.
1:37 Economical Custom Light Stage: A key innovation is a custom-built light stage, significantly more economical than existing solutions. It features 16 high-end machine vision cameras recording at 72 FPS and 40 custom, bright LED modules for single-light illumination patterns.
2:04 New Dataset Creation: Using this setup, a new dataset of multi-view light stage recordings of human heads under various expressions and illuminations was created.
2:15 Superior Performance in Reenactment: Comparisons against state-of-the-art methods (like reliable Gaussian avatars) demonstrate that BecomingLit produces much more accurate and realistic self-reenactment results.
2:41 Generalization to Unseen Illumination: The method excels at generalizing to novel illumination conditions, including environment maps, where baseline methods struggle, producing renderings closer to ground truth.
3:14 Ablation Studies: Ablation studies highlight the importance of the neural BRDF for modeling global illumination effects (avoiding a "plastic-like" appearance) and expression-dependent features for capturing fine wrinkles and geometric details.
3:50 Real-World Application: The avatars can be easily animated from monocular videos and relit with novel environments, showcasing practical application potential.