This video provides a deep dive into the unique characteristics and challenges of serving streaming Automatic Speech Recognition (ASR) models, contrasting them with traditional batch ASR and Large Language Models (LLMs). Focusing on Kyutai's open-source Moshi model and its Rust-based inference server, the presenter details its full-duplex architecture, which utilizes a lightweight Mimi encoder to generate semantic audio tokens for a transformer that outputs text in real time. Experimental results demonstrate efficient GPU memory management, with allocation occurring at startup and remaining stable for numerous concurrent, long-duration streams due to a "KV cache eviction" strategy. However, sending audio faster than real-time is identified as a critical issue leading to unbounded CPU memory growth. Economically, the Kyutai Moshi deployment is shown to be significantly cheaper than commercial alternatives, though streaming ASR is inherently more costly than batch ASR due to continuous GPU memory utilization. The video concludes by highlighting the comparative simplicity of ASR's uniform requests versus LLMs, balanced by the complexities of maintaining long-duration stateful connections and meeting strict real-time latency demands.
Summarizing Streaming Speech Recognition Model Serving
0:00 Introduction to Streaming ASR: The video introduces streaming Automatic Speech Recognition (ASR), defining it as real-time transcription within 1-2 seconds, contrasting it with batch ASR. It highlights the inefficiencies of previous attempts to make OpenAI's Whisper a streaming model and introduces the main topic: Kyutai's recently released Moshi model, primarily designed for real-time dialogue and its open-source Rust inference server for efficient streaming ASR.
1:19 VoiceWriter Promotion: The speaker briefly promotes VoiceWriter, a personal project that uses accurate streaming ASR models to help users write faster by speaking, with AI correcting grammar and adding punctuation in real-time.
1:49 Kyutai Moshi Architecture: The Moshi model is described as a full-duplex model capable of simultaneous listening and talking, integrating speech-to-text, text-to-speech, and reasoning into one unified model. For speech-to-text, audio is first processed by Mimi, a lightweight (~90 million parameter) speech encoder, which converts speech into a sequence of semantic audio tokens. Subsequently, a transformer converts these audio tokens into text, generating one text token for every 80-millisecond audio frame (12.5 frames per second), often resulting in blank tokens due to the faster-than-speaking-rate processing.
4:03 Inference Server Scenarios (Quiz): The speaker poses a quiz, asking viewers to predict how CPU RAM and GPU memory would behave in three scenarios: many concurrent users, very long streaming sessions, and users sending audio faster than real-time.
4:45 CPU and GPU Memory Experiments: An experiment setup is detailed where a Kyutai server listens for WebSocket connections, a Python script gradually ramps up to 64 simultaneous real-time audio streams, and a third process monitors memory usage.
5:19 GPU Memory: It was found that GPU memory (around 9GB for a batch size of 64) is pre-allocated at startup and remains stable, regardless of the number of active connections or the duration of the streaming sessions.
5:40 CPU Memory: CPU memory was observed to increase with the number of connections but then stabilized once the maximum server load (e.g., 64 simultaneous streams) was reached, enabling indefinite streaming without memory exhaustion.
6:08 Streaming Server Mechanisms: The server batches audio data from multiple clients for parallel processing on the GPU and maintains a stateful working memory for each connection to remember previously generated tokens and seen audio.
7:20 KV Cache Eviction: A key engineering technique for indefinite streaming is the "KV cache eviction," where the model only needs the last minute or so of audio context, allowing it to clear out and reuse older parts of the KV cache, thus preventing GPU memory exhaustion during long sessions.
8:18 Faster-than-Real-Time Audio Issue: A critical finding was that sending audio at twice the real-time speed causes CPU memory to continuously increase without bound, as the server consumes audio at a real-time rate, leading to buffer overflow and potential server crashes. This is why many commercial ASR API providers prohibit sending audio faster than real-time.
9:21 Cost Analysis: Based on claims of an H100 GPU processing 400 real-time streams and cloud GPU rental costs, the estimated cost for Kyutai Moshi deployment at scale is approximately 1.5 cents per hour of real-time streaming.
9:27 Cost Comparison: This cost is significantly cheaper (about 10 times) than commercial alternatives like Assembly AI's universal streaming service (15 cents per hour).
9:40 Streaming vs. Batch Cost: Despite its efficiency, streaming ASR models are inherently less efficient than batch ASR because an hour of streaming requires holding expensive GPU memory for an entire hour, making them an order of magnitude more expensive than comparable open-source batch transcription models.
Simpler Aspects: Streaming ASR inference is considered simpler than LLM inference (e.g., vLLM) because it typically does not require multi-GPU setups (models are smaller), requests are uniform (leading to predictable memory allocation without constant rebatching or deallocation), and there's no distinction between prefill and decoding phases.
More Complex Aspects: However, it is more complex due to models needing to hold state for potentially hours-long connections (unlike LLMs, which usually finish requests quickly) and having strict real-time latency requirements, particularly for voice agent systems where delays are highly noticeable.
11:40 Conclusion: The video concludes by reiterating the detailed exploration of a production-suitable streaming speech recognition inference server.