Primary Audience: ML Infrastructure Architects, Senior Site Reliability Engineers (SREs), and Distributed Systems Engineers specializing in Large Language Model (LLM) deployment and orchestration.
Abstract
This technical overview details the system architecture of "Dynamo," an end-to-end, Kubernetes-native framework designed for high-performance LLM inference. The architecture addresses the critical trade-off between interactivity and throughput by supporting both aggregated and disaggregated serving models. Key innovations include the "AI Configurator" for simulation-based offline optimization, the "Grove" scheduler for topologically aware pod scaling, and a Rust-based control plane for low-latency request routing.
Central to Dynamo's efficiency is its sophisticated memory management and data transfer layer. It utilizes "Nixle," a high-performance library for KV cache transfer and offloading, and "Model Express" for rapid weight loading via GPU-to-GPU transfers. The system features a KV-aware router that utilizes precise event-based indexing to maximize cache hits. Furthermore, Dynamo incorporates robust fault-tolerance mechanisms, including request-level migration and eventually consistent state synchronization across router replicas, ensuring high availability in dynamic production environments.
System Architecture and Operational Workflow of Dynamo
0:29 Architectural Flexibility: Dynamo is engineered to handle the non-linear Pareto curve of LLM serving by supporting diverse configurations, including disaggregated pre-fill and decode workers, to meet specific latency and throughput SLAs.
4:17 AI Configurator (Pre-deployment): A simulation-based tool that enables offline performance tuning without requiring GPU resources. It generates optimal Tensor Parallelism (TP) settings and parallelism strategies based on target hardware and latency requirements (TTFT/ITL).
6:10 Kubernetes-Native Control Plane: The system utilizes a custom Dynamo Operator and the "Grove" scheduler to manage pod lifecycles. Grove provides topological awareness and allows for independent scaling of pre-fill and decode "pod cliques" within specific network domains.
8:55 Dynamic "Planner" Scaling: An LLM-specific auto-scaler that monitors real-time metrics. It autonomously scales pre-fill workers to address Time to First Token (TTFT) bottlenecks and decode workers to maintain Inner Token Latency (ITL) targets.
10:04 Model Express & Fast Weight Loading: Optimizes cold-start times through in-cluster caching and direct GPU-to-GPU weight transfers, bypassing traditional bottlenecked storage paths when possible.
11:17 Rust-Based Routing & Front-end: The entry point uses Rust for high-concurrency networking. It provides OpenAI-compatible interfaces and executes tokenization before routing requests to optimal workers based on load and KV cache state.
12:55 Engine Agnostic Execution: The worker core remains agnostic to the underlying inference engine (e.g., vLLM, TensorRT-LLM, SG-Lang), providing a common interface for KV events and scaling operations.
13:39 Nixle Data Transfer: A high-performance library utilized for moving KV caches between workers during disaggregated execution and for offloading cache blocks to CPU/host memory to increase cache hit rates.
14:55 Precise KV-Aware Routing: Unlike approximate routing methods, Dynamo uses standard event-based feedback from workers to maintain a global, precise index of cached blocks, significantly reducing redundant pre-fill computations.
15:52 Request-Level Fault Tolerance: Enables sequence migration during execution, allowing a request to move from a failed worker to a healthy one. It also supports early request cancellation across the entire chain to prevent wasted compute.
18:20 High Availability & State Sync: Router state is synchronized across replicas to prevent single points of failure. Future developments focus on process checkpointing and shadow memory to achieve near-instantaneous recovery from hardware or software faults.