This presentation introduces the concept of disaggregated inference for Large Language Models (LLMs), focusing on the architectural separation of "prefill" and "decode" operations. The speaker, Jinga Chen from UCSD's How AI Lab, argues that "goodput" (requests satisfying Service Level Objectives or SLOs) is a more accurate metric than raw "throughput" for evaluating LLM serving cost and user experience. He highlights two key problems with traditional co-located prefill and decode: interference, where compute-bound prefill elongates memory-bound decode latency, and inefficient resource allocation due to inability to tune parallelism separately. The proposed solution, "DisServe," allocates dedicated GPU resources for each phase, demonstrating a significant "goodput" gain (up to 2x per GPU, 7.2x overall). The talk delves into design challenges like efficient KV cache transfer, optimal parallelism configuration using simulation, and combating burstiness with pull-based KV cache transfer. Finally, it discusses the widespread industry adoption of disaggregated serving (e.g., Nvidia Dynamo, Mooncake) and future research directions like fast reconfiguration for true serverless LLM systems.
Summarizing Disaggregated Inference for LLMs
0:04 Introduction to Disaggregated Inference: The talk introduces disaggregated inference, a highly impactful area of ML systems research, as featured in Jensen Huang's GTC keynote.
1:13 Throughput vs. Goodput: The speaker argues that "goodput" (requests meeting specific Service Level Objectives or SLOs) is a superior metric to "throughput" (total requests/tokens per second) for evaluating LLM inference, as it accounts for user experience and effective resource utilization.
4:52 SLO Metrics: Key SLOs for LLMs are defined: TTFT (Time To First Token) and TPOT (Time Per Output Token), which vary based on application (e.g., chatbot vs. programming agent).
9:37 Distinct Characteristics of Prefill and Decode:
Prefill: The initial processing of the entire input prompt; characterized as compute-bound, easily saturating GPUs even with single requests.
Decode: The auto-regressive generation of subsequent tokens; characterized as memory-bound, requiring large batch sizes to saturate compute.
11:09 Prefill & Decode Animation: A visual explanation demonstrates how prefill processes the input to generate KV caches and the first token, while decode generates subsequent tokens one by one using these caches.
14:54 Problem 1: Interference in Hybrid Batching: When prefill and decode requests are batched together (chunkfill), the compute-intensive prefill significantly elongates the latency of the memory-bound decode operations, leading to poor user experience, especially with longer prompts (e.g., 12x longer decode for 1K prompt).
17:50 Problem 2: Inefficient Resource Allocation: Co-locating prefill and decode prevents separate tuning of parallelism strategies (e.g., tensor parallelism for latency, pipeline parallelism for burstiness), leading to suboptimal GPU utilization for different SLO needs.
39:40 DisServe Solution - Disaggregation: The core idea is to separate GPUs into dedicated "prefill workers" and "decode workers," linked by high-speed networks (like NVLink). Requests migrate from prefill to decode after the first token is generated.
40:51 Goodput Gain from Disaggregation: A simple example illustrates how disaggregation can yield a 2x increase in "goodput" per GPU compared to co-located systems, by optimizing resource allocation for prefill and decode phases independently.
High Network Affinity (e.g., NVLink): KV cache transfer overhead is negligible (<0.1% of latency) due to high bandwidth (120-150 GB/s), taking ~1ms for a 1K context token.
Low Network Affinity (e.g., InfiniBand between nodes): Pipeline parallelism is used across nodes to transfer only activations, while KV cache transfer remains within high-affinity intra-node connections.
50:59 Optimal Parallelism Strategy: A brute-force simulation algorithm (running in under 2 minutes) helps identify optimal tensor parallelism (TP) and pipeline parallelism (PP) configurations for prefill and decode workers based on workload characteristics and SLOs.
54:30 Combatting Burstiness with Pull-Based KV Cache Transfer: Instead of prefill "pushing" KV cache to decode (which can saturate decode's memory), decode "pulls" the KV cache from prefill when it's ready, allowing it to manage memory more effectively and combat burstiness.
56:43 Reducing Pipeline Bubbles: Simple chunking of requests is used to reduce pipeline bubbles in pipeline parallelism, though more advanced techniques exist.
57:10 Evaluation Results: DisServe achieved up to 7.2x goodput improvement over VLM alone by satisfying strict SLOs on various datasets (ShareGPT, HumanEval, LongBench).
1:00:01 Industry Adoption: Major companies (e.g., Nvidia with Dynamo, Mooncake) and serving engines (VLM, Sglang) have adopted disaggregated serving.
1:00:46 Nvidia NIKSL: Nvidia Inference Transfer Engine (NIKSL) is highlighted as a tailored library for low-latency, interconnect-agnostic multi-node data transfer of KV caches.
1:01:31 KV Cache Aware Routing: The importance of smart global routers that direct requests to prefill instances based on KV cache hit potential to minimize TTFT.
1:04:50 Future Work (2025): The speaker's wish list includes fast reconfiguration of system parallelism based on changing workloads and the development of "true serverless LLM systems" to further reduce serving costs.
1:06:41 UCSD Lab Projects: Mentions UCSD's CSE234 LLM system class, Game Arena (evaluating LLMs playing games), and Dinosaur (efficient reasoning model serving using dynamic resource allocation and PD aggregation for chain-of-thought).
1:08:48 Conclusion: Reaffirms that goodput is the critical metric, PD disaggregation is highly effective (7.2x gain), and the technique is widely adopted.