Okay, here is the abstract and summary for the provided transcript about the "Sheep" method.
Abstract:
This presentation introduces Sheep (Self-supervised Head Geometry Predictor), a novel method for reconstructing expressive and geometrically accurate 3D head models from single 2D images. The core idea involves using an encoder (a Vision Transformer) to predict parameters for a 3D Morphable Model (3DMM) mesh based on identity, expression, and pose features extracted from input images. This mesh then controls a 2D Gaussian Splatting renderer, which reconstructs the input view. Training is self-supervised, primarily using a photometric loss between the rendered and input images, augmented by a 2D landmark loss and a novel regularizer that enforces geometric consistency (normals and depth) between the rendered output and the predicted mesh in the facial region. Sheep achieves state-of-the-art results on the Now benchmark compared to other 2D-supervised methods, demonstrates superior handling of facial expressions without exaggeration compared to prior works like Smirk and Emoka, shows high temporal stability despite frame-by-frame processing, and uniquely estimates neck and torso pose.
Sheep: Self-supervised 3D Head Geometry Prediction
0:00:04 Goal: Introduce "Sheep," a method to predict expressive and accurate 3D head geometry from a single 2D input image using self-supervision and 2D Gaussian splatting.
0:00:33 Core Pipeline: An input image goes through an encoder to predict a 3DMM mesh. This mesh controls a Gaussian Splatting renderer to create a reconstruction, which is compared to the input via photometric loss for self-supervised training.
0:00:58 Training Setup: The system trains on pairs of source and target images from 2D videos. A Vision Transformer extracts identity/lighting features from the source and expression/pose features from the target.
0:01:21 Mesh & Gaussian Prediction: Shape, expression, and pose parameters determine the 3DMM head mesh. DinoV2 features from the source image are used by a Gaussian regressor to predict 3D Gaussians.
0:01:34 Rigging and Rendering: The predicted Gaussians are rigged (attached) to the 3DMM mesh and shaded using predicted environment lighting. They are then rasterized to reconstruct the target image.
0:01:50 Loss Functions: Training uses photometric loss (comparing rendered vs. target image) and a 2D landmark loss based on the predicted mesh.
0:01:59 Novel Geometric Regularizer: A key contribution is a regularizer encouraging the rendered normals and depth in the face region to match those of the underlying predicted mesh, crucial for high performance.
0:02:13 State-of-the-Art Performance: Sheep achieves leading non-metrical error on the "Now" evaluation benchmark against other methods using only 2D supervision.
0:02:25 Qualitative Results: The method produces realistic, expressive, and geometrically accurate head meshes.
0:02:36 Neck and Torso Pose: Sheep is highlighted as the first method of its kind to accurately estimate both neck and torso pose.
0:02:48 Expression Handling: Compared to methods like Smirk and Emoka, Sheep avoids over-exaggerating facial expressions and shows better performance in emotion prediction on the AffectNet dataset.
0:03:06 Temporal Stability: Sheep's predicted meshes exhibit high temporal stability, outperforming TokenFace even though Sheep processes frame-by-frame and TokenFace uses temporal smoothing.
0:03:41 Geometric Evaluation: The method shows strong quantitative results in geometric evaluations using expressive point clouds from the Nübble dataset.