This video introduces PercHead, a unified model for high-fidelity 3D head reconstruction from a single image and subsequent semantic editing. The model's architecture employs a dual-branch 2D encoder with a transformer-based decoder to lift features into 3D, rendered using Gaussian Splatting. A key innovation is its training, which relies on a novel perceptual supervision strategy using foundation models DINOv2 and SAM2.1, rather than traditional loss functions. This approach enables state-of-the-art reconstruction quality and remarkable robustness, particularly with extreme input angles like side profiles. The base model can be seamlessly adapted for editing by swapping its encoder, allowing for disentangled control over geometry via segmentation maps and appearance (style) via reference images or text prompts. The presentation demonstrates these capabilities, including zero-shot text-guided editing and an interactive web application for intuitive 3D head creation and modification.
PercHead: A Breakthrough in Single-Image 3D Head Reconstruction and Editing
0:00 Dual-Purpose Model: PercHead is introduced as a method that performs two key tasks: reconstructing a high-quality 3D head from a single 2D image and enabling flexible, semantically-controlled 3D editing.
0:29 Reconstruction Architecture: The model uses a dual-branch 2D encoder (combining a frozen DINOv2 and a lightweight Vision Transformer) to extract features. A transformer-based decoder then lifts these features into 3D space, and the final head is rendered using Gaussian Splatting.
1:10 Novel Perceptual Supervision: A core innovation is the training method, which forgoes traditional losses (like L1) in favor of supervision from powerful foundation models—DINOv2 for visual understanding and SAM 2.1 for segmentation-oriented understanding. This results in superior geometric and appearance fidelity.
1:35 Disentangled 3D Editing: The model can be adapted for editing by replacing the encoder. It uses a segmentation map to control geometry and a CLIP encoder to control style (from a reference image or text prompt), naturally separating shape from appearance.
1:57 Robustness to Extreme Views: PercHead demonstrates exceptional performance even with challenging inputs, such as a side view, where it must realistically hallucinate large unseen regions while maintaining the subject's identity.
2:11 Outperforming Baselines: In comparisons, PercHead maintains visual quality and identity across diverse viewpoints, while competing methods often produce artifacts, distort identity, or fail to generate plausible novel views from challenging angles.
2:43 Application to Video: By applying the reconstruction process frame-by-frame, the model can generate a consistent 3D avatar from video, preserving expressions and identity over time without explicit temporal modeling.
3:13 Zero-Shot Text-Guided Editing: The model supports text-based style editing without being explicitly trained for it. By leveraging CLIP's aligned vision-text encoder, it can interpret prompts to change attributes like hair texture, color, and even perceived age.
3:32 Interactive Web Application: A demo showcases a user-friendly GUI where a user can upload a photo, edit its segmentation map by drawing, and apply different styles using either a reference image or text prompts to intuitively create and modify 3D heads.