Reviewer Group: ML Infrastructure (MLInfra) and Systems Architecture Specialists

This topic is best reviewed by Senior ML Infrastructure Architects and Distributed Systems Engineers. These professionals are responsible for the orchestration, scaling, and cost-optimization of LLM and Diffusion model deployments. They focus on hardware utilization, latency-sensitive Service Level Objectives (SLOs), and the co-evolution of model architectures and system backends.

Abstract

This technical presentation by Hao Zhang (UC San Diego) details the architectural paradigm shift in AI inference from 2025 into 2026. The core of the talk addresses the transition from "continuous batching" to "disaggregated prefill and decode (PD)" serving, which optimizes "goodput"—the measure of throughput that adheres to specific latency budgets (TTFT and TPOT).

The second half explores emerging frontiers: Attention-FFN Disaggregation (AFD) and Video Diffusion (DIT). AFD proposes splitting internal transformer modules to maximize utilization in Mixture-of-Experts (MoE) models, utilizing "ping-pong" pipelining to mask communication overhead. The discussion concludes with the systemic challenges of Video Diffusion Transformers, which require processing massive sequence lengths (115k+ tokens) across iterative diffusion steps, necessitating next-generation inference engines like "FastVideo" to move toward real-time 4K generation.

Inference Systems Evolution: Disaggregation and Video Diffusion

• 0:00 – Introduction: Hao Zhang (UCSD/Disserv) provides a roadmap for the talk, focusing on the 2025 trend of Prefill/Decode disaggregation and 2026 projections for internal module splitting and video workloads.
• 1:41 – The "Goodput" Metric: Effective inference is defined not just by raw throughput, but by "goodput"—throughput that satisfies two primary SLOs:
  • TTFT (Time to First Token): Critical for user experience in chatbots.
  • TPOT (Time per Output Token): Critical for high-speed summarization and reading speed.
• 4:43 – Continuous Batching vs. Disaggregation: Standard continuous batching suffers from interference; a new prefill request (compute-bound) can spike the latency of an ongoing decode request (memory-bound). Disaggregation eliminates this by moving requests between dedicated "Prefill" and "Decode" workers.
• 7:44 – Strategic Partitioning: Disaggregation allows for "Divide and Conquer" optimization. Prefill instances can use Tensor Parallelism to minimize TTFT, while Decode instances utilize Data Parallelism and larger batch sizes to maximize TPOT.
• 9:17 – Case Study: 2P1D Allocation: Profiling shows that allocating two prefill workers to one decoder worker (2P1D) can double the goodput per GPU compared to co-located systems by balancing the specific resource demands of the workload.
• 11:12 – The XPYD Equation: The core challenge of modern inference is solving for placement (how many P vs. D units) and communication (efficient KV-cache transfer between heterogeneous hardware).
• 12:55 – Industry Milestones (2025):
  • DeepSeek-V3: Successfully embraced PD disaggregation with specialized parameters.
  • NVIDIA Dynamo: The current state-of-the-art production implementation, featuring KV-aware routers, GPU planners, and low-latency transfer layers.
• 17:06 – Trend 1: Attention-FFN Disaggregation (AFD): The next evolution involves splitting the attention module from the FFN/MoE module within a single layer. This is particularly effective for MoE models where expert parallelism can be scaled independently from attention replicas.
• 19:21 – The Ping-Pong Pipeline: To mitigate the "scary" per-layer communication overhead of AFD, systems use fused communication (combining AFD moves with existing MoE all-to-all) and "ping-pong" pipelining to overlap micro-batch computation with hidden state transfers.
• 22:55 – Trend 2: Video Diffusion (DIT): Video generation is currently prohibitively expensive (approx. $10/minute of video). Unlike LLMs, Diffusion Transformers (DIT) must run the same stack 50–100 times per generation across multiple diffusion timesteps.
• 25:50 – The 115k Token Challenge: In models like Hunyuan Video, a 5-second 720p clip results in a sequence length of 115k tokens. Over 80% of compute time is spent on quadratic attention, making current single-GPU generation (16 minutes on an H100) impractical for production.
• 27:18 – FastVideo and Real-Time Goals: The "FastVideo" engine aims to optimize attention kernels and memory layout to achieve real-time 1080p and 4K video generation in 2026 by converging diffusion techniques with large-scale language model inference architectures.