GAF: Gaussian Avatar Reconstruction from Monocular Videos via Multi-view Diffusion
0:00 Introduction: GAF is introduced as a new method for reconstructing avatars from monocular videos, using multi-view diffusion to create realistic 3D models.
0:18 Challenges: Existing methods struggle with limited views in monocular videos, leading to artifacts in novel view renderings.
0:37 Solution: GAF utilizes multi-view head diffusion priors to enhance reconstruction, especially in less observed regions.
0:44 Multi-view Diffusion Model: A normal map conditioned multi-view head diffusion model generates consistent output images from a single view, using normal maps from flame head tracking for precise viewpoint control.
1:06 3D Attention Blocks: These blocks ensure 3D consistency across views and preserve identity and appearance from the input image.
1:21 Generation Results: The model produces identity-preserved, view-consistent multi-view portrait images from a single input.
1:40 Gaussian Avatar Reconstruction: Flame tracking provides head geometry, and an animatable Gaussian splatting representation is optimized.
2:02 Regularization: Four random viewpoints are sampled to render novel view images, regularized using the multi-view head diffusion model to generate pseudo ground truths.
2:19 Latent Upsampler: Enhances facial details before decoding the denoised latent into an RGB image.
2:26 Evaluation: GAF outperforms existing methods on the NeRSemble dataset, particularly in reconstructing less observed side-face regions and producing consistent renderings from holdout views.
3:00 Real-World Application: Demonstrates high-fidelity avatar reconstructions from monocular videos captured on smartphones.
3:44 Ablation Studies: Studies on different diffusion priors highlight the effectiveness of multi-view diffusion and normal map conditioning. Using pseudo-image ground truths and a latent upsampler further improves photorealism and face details.
4:27 Conclusion: GAF effectively addresses the limitations of previous methods, offering improved photorealism and consistency in novel view synthesis.