0:00 Introduction: Speculative decoding is introduced as a method to accelerate and reduce the cost of inference in Large Language Models (LLMs) without sacrificing quality.
0:00 Background: LLMs generate output one token (word or part of a word) at a time, making inference slow. Larger models are slower due to their size.
0:00 Observation 1: Some tokens are easier to generate than others. Smaller models can approximate larger models for easier tokens, while larger models excel at harder tokens.
0:00 Observation 2: LLM inference is often bottlenecked by memory bandwidth, not computational power, leaving spare computational resources.
0:00 Speculative Execution: Inspired by speculative execution in CPUs, the method computes multiple tokens in parallel using a fast approximation function, f*(X), alongside the slower, main function, f(X).
0:00 Speculative Sampling: An extension of speculative execution to stochastic settings (where tasks have probabilities), allowing probabilistic acceptance/rejection of approximations.
0:00 Speculative Decoding: Applies speculative sampling to LLMs, efficiently calculating a token and subsequent tokens in parallel, maintaining the original output distribution.
0:00 Implementation: Using smaller, off-the-shelf models or heuristics as fast approximations for larger models yields significant speed improvements (e.g., 3x for an 11B parameter model using a 60M parameter model).
0:00 Adoption and Future: Speculative decoding has been widely adopted in the industry, with various optimizations and applications. The need for efficient inference is growing, and further advancements are anticipated.
0:00 Impact on AI Overviews: Speculative decoding significantly optimizes AI Overviews in Google Search, enabling faster results while maintaining quality.
0:00 Acknowledgements: The authors acknowledge the contributions of numerous individuals and teams from Google Research, Google DeepMind, and Google Search.