Senior AI Infrastructure Architect & Systems Research ScientistExpertise: Distributed Systems, GPU Orchestration, Large-Scale Inference Optimization, and Multimodal Architecture.
Abstract:
This presentation introduces Cornserve, an open-source, scalable system designed for the efficient serving of "any-to-any" multimodal AI models. Current inference frameworks (like vLLM or SGLang) often treat multimodal models as point solutions, leading to suboptimal resource utilization across varying workloads. Cornserve addresses this by implementing component-level disaggregation, separating encoders, LLMs, and generators into distinct, independently scalable units.
The system's core innovation is an automated planner that treats deployment as a multicommodity network design problem. By profiling specific model graphs and workload distributions, the planner determines the optimal balance between monolithic and disaggregated deployment to maximize throughput or minimize latency. Architectural highlights include a "Record and Replay" mechanism for stateless task execution and a distributed runtime built on Kubernetes that reuses existing high-performance inference kernels. Evaluations on Qwen-Omni and InternVL models demonstrate up to a 3.8x throughput improvement over monolithic baselines and the elimination of Out-Of-Memory (OOM) constraints for complex omni-modal configurations.
Cornserve: System Architecture and Performance for Multimodal Inference
0:46 Project Introduction: Cornserve is presented as a system for "any-to-any" model serving, available as an open-source project at cornserve.ai. It focuses on the growing complexity of models that handle text, image, audio, and video interchangeably.
4:42 Transition to Any-to-Any: The speaker traces the evolution from LLMs to VLMs (Vision-Language Models) and finally to Omni models (e.g., GPT-4o, Qwen 2.5 Omni). These models introduce "generators" (e.g., audio generators) alongside traditional encoders and LLM backbones, creating high request and component heterogeneity.
8:03 Serving Evolution and PD Disaggregation: The fundamental generation process is split into pre-fill (compute-bound) and decode (memory-bound). The speaker explains Prefill-Decode (PD) disaggregation as a prerequisite for understanding why static monolithic deployments fail to scale multimodal workloads.
11:12 The Failure of Point Solutions: Analysis shows that for models like InternVL, the "best" deployment (Monolithic vs. Encoder-Prefill/Decode disaggregation) varies wildly depending on input resolution and token counts. Existing systems lack the flexibility to adapt to these shifts.
15:22 Component Heterogeneity: Profiling reveals that components like audio generators are often significantly slower than the LLM backbone. In a monolithic setup, the entire system is bottlenecked by the slowest component, wasting high-performance GPU resources.
17:39 Cornserve System Overview: The architecture consists of a Gateway, an Automated Planner (containing a profiler and solver), and a Distributed Runtime. The runtime manages "Task Managers" and "Task Executors" across a Kubernetes cluster.
18:57 Task Abstraction: Workloads are divided into Unit Tasks (the smallest unit of scaling/execution, e.g., an image encoder) and Composite Tasks (user-defined Python compositions of unit tasks). This allows for resource sharing, such as sharing a single vision encoder across different model variants (e.g., Gemma 3 4B and 12B).
21:05 The Planner and Solver: The planner performs path analysis on request types and utilizes a multicommodity network design solver to determine the optimal number of replicas and routing paths. This automates the decision of whether to disaggregate or colocate components based on target workload.
24:18 Record and Replay Mechanism: To maintain statelessness and enable parallel execution of non-dependent modalities (e.g., processing audio and video simultaneously), Cornserve uses a symbolic execution "record" phase at the gateway followed by a "replay" at the dispatcher level.
28:51 Implementation Details: The system comprises ~15,000 lines of Python on Kubernetes. It reuses vLLM for the LLM executor but allows for the integration of any engine (SGLang, TensorRT-LLM) due to its modular design.
30:53 Evaluation - Multi-modal Output: Using Qwen 2.5 Omni on A100 GPUs, Cornserve achieved a 3.09x to 3.81x throughput increase over monolithic Transformer baselines. It also successfully served Qwen 3 Omni, which crashed in monolithic setups due to OOM errors.
33:01 Evaluation - Multi-modal Input: In InternVL tests, the Cornserve planner matched or surpassed the best-known human-configured deployment strategies by dynamically routing requests between dedicated encoders and monolithic replicas.
39:06 Future Trends and Edge Potential: The speaker notes that as model components shift (e.g., from diffusion-based to convolution-based audio generators), flexible orchestration becomes even more critical. There is potential for "dual setups" where encoding and generation occur locally on-device while the LLM backbone runs remotely.