Meshtron: A Scalable Solution for High-Fidelity 3D Mesh Generation

• Problem:
  • 0:00 Traditional methods are lacking: Hand-designed algorithms like Marching Tetrahedra and Marching Cubes produce dense meshes with unorganized topology, leading to poor quality and limited usefulness.
  • 0:00 Scalability issues: Existing data-driven methods, while promising, are limited to generating meshes with a low face count (up to 1.6K) and low spatial resolution (128-level).
• Solution:
  • 0:00 Meshtron: A data-driven, autoregressive model that generates intricate, artist-like meshes with up to 64K faces at 1024-level coordinate resolution.
  • 0:00 High face count and resolution: Over an order of magnitude higher face count and 8x higher coordinate resolution compared to existing methods.
• Technical Details:
  • 0:00 Mesh as a sequence of tokens: Meshes are converted into a sequence of tokens representing triangle faces, vertices, and coordinates, sorted from bottom to top.
  • 0:00 Hourglass Transformer architecture: Reduces computational cost and memory usage by merging tokens internally, significantly speeding up the generation process.
  • 0:00 Efficient token allocation: The model allocates more computational resources to difficult-to-generate tokens, improving efficiency and modeling accuracy.
  • 0:00 Sliding window attention: Maintains a fixed-length context window, leading to constant memory cost and token throughput, preventing slowdown with larger meshes.
• Advantages:
  • 0:00 2.5x faster token throughput: Achieves faster training and inference compared to conventional models.
  • 0:00 Over 50% memory saving: Reduces memory consumption during both training and inference.
  • 0:00 High controllability: Accepts control inputs like point clouds, face count, quad ratio, and creativity level, enabling versatile mesh generation.
• Applications:
  • 0:00 Stand-alone remesher: Improves the quality of existing meshes.
  • 0:00 Text-to-3D/Image-to-3D: Generates artist-grade meshes from scratch when combined with text-to-3D or image-to-3D models.
• Conclusion:
  • 0:00 Unprecedented resolution and fidelity: Meshtron generates meshes of complex 3D objects with high detail, closely resembling those created by professional artists.
  • 0:00 Potential for various industries: Opens doors for more realistic 3D asset generation in animation, gaming, and virtual environments.
  • 0:00 Reference for future research: Its general and extensible architecture can serve as a foundation for future mesh generation research and applications.
• Resources:
  • 0:00 Links provided: Includes links to a video, whitepaper, and demo video for further information.