This video features Perry Jang from UCSD's How AI Lab discussing "Fast Video," a project aimed at accelerating large video diffusion models. He highlights that current state-of-the-art models are significantly bottlenecked by the iterative nature of the diffusion process and the computational cost of attention mechanisms, often taking minutes to generate mere seconds of video even on high-end GPUs. Jang outlines various acceleration techniques, including distillation (reducing diffusion steps), feature caching, and particularly, sparse attention. He elaborates on Video Sparse Attention (VSA), a novel approach that leverages the inherent sparsity of attention scores and employs a hierarchical architecture (coarse and fine branches) combined with token reordering to achieve significant speedups during both training and inference while
Error: int() argument must be a string, a bytes-like object or a real number, not 'NoneType' maintaining quality. The presentation emphasizes the project's goal of building a unified and efficient engine for video generation, pushing towards real-time inference.
Summarizing the Transcript: Accelerating Video Diffusion Models
0:00:20 Introduction to GPU Mode and Perry Jang: The host introduces Perry Jang from UCSD's How AI Lab, who will discuss "Fast Video" and accelerating large video diffusion models, shifting focus from typical language model (LLM) topics.
0:02:28 Video Generation Models are Taking Off: Video generation models are rapidly improving in quality (e.g., Google's V3) and are seen as potential "world models" for embodied AI and agent tasks, though their market share isn't yet as large as LLMs.
0:03:44 How Video Diffusion Transformers (DiT) Work: Explains the core process: a diffusion transformer denoises a noisy input conditioned by a text prompt and time step (20-50 iterations). 3D video data is flattened into a 1D sequence, and a decoder produces the high-resolution output.
0:05:16 High Inference Cost of Video DiT: Using Huan video (5-second, 720p) as an example, a single forward pass of the diffusion transformer takes 19 seconds on an H100 GPU, resulting in 16 minutes for 50 time steps. This cost is dominated by long causal attention (over 85% of FLOPs).
0:06:50 Two Main Bottlenecks: The iterative nature of diffusion and the disproportionate amount of attention FLOPs.
Distillation: Methods like Progressive Distillation and Consistency Models can reduce steps by 5-10x. However, distillation training is unstable, and open-source implementations for video generation are limited.
Feature Caching: Leverages heuristics from neighboring diffusion time steps to reuse intermediate feature maps, accelerating inference by 2x without requiring retraining.
0:11:51 Fast Video's Current Implementations: Incorporates PCM (a distillation method) and TCH (a feature caching method) and aims for a unified inference framework for easier comparison and integration of new techniques.
0:12:35 Reducing Attention FLOPs (Sparse Attention): Attention scores are generally very sparse, meaning only a few indices have high scores. If these patterns can be predicted, computation can be saved.
0:17:39 Hardware Constraints and Block Sparse Attention: To be truly efficient on GPUs, sparse attention needs to be "block sparse" (dense or completely empty blocks) to avoid wasted computation from "mixed" blocks.
0:23:09 Token Reordering for Efficient Sparse Attention: Reordering sequences (e.g., mapping 3D video cubes like 4x4x4 to contiguous 1D blocks) allows for the creation of efficient block sparse patterns. This ensures that a whole "cube" (or block) either attends or doesn't.
Video is partitioned into 4x4x4 cubes (64 tokens per block).
Coarse Branch: Performs average pooling on QVA (Query, Key, Value) to get lightweight cube-level representations, providing global context.
Fine Branch: Computes token-level attention with sparsity, picking critical blocks based on the attention map from the coarse branch.
Gating Mechanism: Balances contributions from coarse and fine branches.
0:33:13 VSA Performance: Achieves speedups of ~6-7x for attention (close to theoretical 8x for 87.5% sparsity), which is significant for huge sequences.
Data-dependent trainable sparsity (VSA) consistently outperforms fixed patterns and full attention.
Including global information from the coarse branch is crucial.
A trade-off exists between block size (smaller block size = more expressive attention pattern, but slower kernel; larger block size = less expressive, but faster kernel). The optimal choice balances loss and latency.
0:43:17 VSA Scalability: VSA matches or outperforms full attention in terms of loss per FLOPs, especially at larger scales (e.g., 410M parameters), demonstrating better scalability.
0:44:56 Optimal Sparsity: Depends on training budget and sequence length; more overtraining allows for lower sparsity (more compute). For longer sequences, a higher effective sparsity might be optimal due to undertraining.
0:48:40 Fine-tuning Existing Models with VSA: A method of gradually converting a full attention model to VSA is proposed, starting with zero-initialized coarse branch weights and gradually decreasing the top-K sparsity during training.
0:51:32 Broader Picture of Video Generation Acceleration: Distillation offers the largest speedup (5-10x) but is challenging due to instability and fragmented open-source code. Feature caching and sparse attention offer 2x speedups. Sparse attention also benefits training speed.
0:52:42 Future Directions: Quantization (e.g., FP8 linear) and fused kernels are other potential areas.
0:53:23 The Fast Video Project's Goal: To build a unified, efficient training and inference engine for video generation models, supporting various models, distillation techniques, and sparse attention methods to reduce inference time from minutes to real-time.
0:55:11 Diffusion vs. LLM Inference: Diffusion is currently compute-bound (LLMs can be memory-bound). Diffusion models require enormous sequence lengths for even short videos (100k tokens for 5-second 720p), posing a unique challenge for scaling to longer videos. Diffusion pipelines involve multiple models (encoder, VAE, diffusion) requiring complex sharding. The fields are converging, with lessons learned from LLM infrastructure potentially applicable.