3D Gaussian Splatting: A Method for High-Quality, Real-Time Novel View Synthesis
0:00:14 Introduction: George Drettakis is introduced as a leading figure in the rendering field, known for his work on Gaussian Splatting.
0:01:53 What is Gaussian Splatting?: It's a method for real-time novel view synthesis using 3D Gaussians as a rendering primitive. The talk will cover the history and development of this technique.
0:02:23 Motivation: Drettakis's goal is to generate realistic 3D environments for real-time navigation, initially using path tracing.
0:02:44 Traditional Computer Graphics Pipeline: The process involves creating geometry, assigning materials, placing lights and cameras, and performing light transport calculations.
0:03:41 The Challenge of Creating 3D Scenes: Drettakis's interest lies in capturing 3D scenes from images or videos for graphics applications.
0:05:00 Novel View Synthesis: Also known as image-based rendering, involves using calibrated cameras and approximate geometry to reproject pixels and blend them for novel views.
0:05:57 Early History (1997): Paul Debevec's work on image-based modeling and rendering, and the golden age of point-based rendering with Q-Splat.
0:07:31 The Point Brothers: Hans-Peter Pfister, Matthias Zwicker, and Mark Pauly developed the first work on surfels, the basis of point-based rendering.
0:08:04 Early 2000s: Exploration of point-based rendering for complex scenes and image-based rendering for capturing objects like trees.
0:10:01 2007: The advent of structure from motion and multi-view stereo significantly improved the automatic generation of 3D models from images.
0:11:17 2013: Using multi-view stereo meshes for image-based rendering, with per-view information to compensate for geometry errors.
0:12:19 2014: Deep learning starts to be applied to novel view synthesis, learning blending weights for improved results (Deep Blending).
0:16:42 Recent Past: Drettakis's ERC Advanced Grant funded research on differentiable rendering, removing multi-view stereo meshes, and exploiting GPU rasterization.
0:18:21 2021: Development of point-based neural rendering, using rasterization and a CNN for high-quality results, but still slow.
0:25:12 2022: A "niche" project on mirror-like surfaces led to the removal of per-view information, a crucial step towards Gaussian Splatting.
0:27:49 2023: 3D Gaussian Splatting is introduced, using 3D Gaussians as an efficient and effective 3D representation.
0:28:45 3D Gaussians: Extension of 2D surfels to 3D, with position, covariance matrix, opacity, and spherical harmonics for color.
0:29:53 Rendering: Efficient tile-based rendering and sorting on the GPU, developed by Bernhard Kerbl.
0:31:27 Optimization: Optimization of all Gaussian parameters using PyTorch and Adam, with techniques for adding and removing points.
0:31:53 Evaluation: Comparison with state-of-the-art methods like NeRF and Instant-NGP, showing significant improvements in speed and quality.
0:35:53 Uptake and Applications: Gaussian Splatting has been widely adopted in various fields, including e-commerce, visual effects, and virtual reality.
0:36:57 Follow-up Work: Memory reduction, solutions for animated scenes, and generative models using Gaussian Splatting.
0:37:16 Large Scenes: Handling large scenes with a hierarchical subdivision approach, allowing real-time navigation of city-scale captures.
0:39:17 Future Work: Drettakis's new ERC project aims to develop a new 3D representation compatible with physically based rendering and generative neural rendering.
0:40:06 Relighting: Using diffusion models as priors for relighting Gaussian Splatting scenes.
0:43:39 Challenges: Finding a representation that is both good for rendering and optimization, and developing fast and accurate differentiable renderers.
0:44:14 Conclusion: Novel view synthesis is now usable in many applications, but creating 3D content still requires significant work.
0:45:16 Take-Home Lessons: Importance of balancing new tools with existing knowledge, exploiting strengths, acknowledging errors, and having fun.
0:47:50 Q&A: Discussion on explicit vs. implicit representations, the future of meshes, physically based rendering, and dynamic scenes.