This presentation outlines recent and forthcoming architectural advancements within the Dynamo framework aimed at enhancing support for complex machine learning model serving, particularly in the realm of multimodality and multi-stage inference pipelines.
Key developments include the integration of first-class support for the prevalent multimodal pattern: Encode, Pre-fill, Decode (EPD). To optimize performance, an Embedding Cache Module (ECM) is being introduced to bypass costly encoder re-execution, leveraging concepts similar to Radix trees used for Key-Value (KV) cache pre-fill. Furthermore, Dynamo is evolving to facilitate the definition and serving of intricate inference pipelines, where multiple inputs (e.g., video and text) pass through sequential processing stages (multiple encoders, diffusion models) to generate a final output. These pipelines can now be architected and deployed using pure Python definitions wrapped by Dynamo’s Rust core, enabling flexible and efficient routing via Python request paths and integrated discovery services.
Dynamo Infrastructure Advancements for Multimodal and Pipelined Inference
0:09 Multimodal Support: Dynamo is actively adding first-class support for multimodal models, focusing on the standard Encode, Pre-fill, Decode (EPD) architectural pattern.
0:31 EPD Routing: Work is underway to implement dedicated routing support within Dynamo specifically designed for EPD architectures.
0:48 Embedding Cache (ECM): Support has been added for an embedding cache designed for EPD models, specifically addressing the cost of encoding multimodal inputs (like images).
0:54 Performance Optimization: The ECM functions similarly to how Radix trees are utilized for KV cache during the pre-fill stage, allowing the system to avoid re-running the encoder when embeddings are cached.
1:01 Development Timeline: The Embedding Cache Module (ECM) is currently in early development and is projected to be integrated into Dynamo within the next few months.
1:31 Complex Pipeline Architecture: Dynamo now facilitates the definition of complex inference pipelines, citing an example scenario involving a pipeline that accepts video and text inputs, processes them through multiple encoders and a diffusion model, and outputs an upscaled video.
2:06 Model Stage Definition: Dynamo enables the definition of these multi-stage models in pure Python. These Python definitions are wrapped within Dynamo's core Rust framework.
2:23 Serving Mechanism: The deployment and serving of these complex pipelines are managed by specifying a Python request path. This path dictates the flow of execution, tracing calls through various stages and utilizing Dynamo’s internal discovery service for component identification and routing.
2:39 Future Engagement: Interested parties are directed to the Dynamo GitHub repository and upcoming roadmap for further details.