This presentation from the Google AI Edge team details significant advancements in their platform for deploying language and machine learning models on mobile and web devices. The talk introduces Light RT, a new runtime designed to simplify the developer experience for hardware acceleration on GPUs and NPUs. Key announcements include expanded model support through a Hugging Face community, advanced customization via INT4 quantization, and a streamlined fine-tuning workflow. The team also unveils powerful new capabilities through dedicated SDKs for on-device Retrieval Augmented Generation (RAG) and Function Calling (Tool Use). Infrastructure improvements are highlighted by the introduction of "ML drift," a re-architected GPU delegate delivering up to 12x performance gains, and simplified integration with NPU hardware from partners like Qualcomm and MediaTek. The presentation concludes by showcasing real-world partner applications that leverage these tools for tasks like real-time deepfake detection and audio processing.
Unlocking On-Device AI: A Deep Dive into Google AI Edge Updates
0:00:38 - The Value of On-Device AI: The presentation outlines the core benefits of running ML on the edge: low latency for real-time applications, offline capability, enhanced user privacy, and reduced server-side costs for developers.
0:01:57 - Expanded Model Support: Google has launched the "Lighter t" community on Hugging Face, providing developers with pre-converted, edge-ready language models like Gemma and DeepSeek that can be used out of the box.
0:03:00 - Advanced Model Customization: A clear workflow is provided for customizing models. Developers can use standard tools like Hugging Face Transformers for fine-tuning, then use Google AI Edge tools to convert the model for on-device inference.
0:04:13 - INT4 Quantization for Performance: The AI Edge Quantizer now supports INT4 post-training quantization, a powerful technique to reduce a model's size by up to 8x and significantly speed up inference with minimal impact on quality.
0:05:41 - New Developer Tools: The team introduced two key tools: Model Explorer for visualizing and debugging model architectures, and AI Edge Portal, a new cloud service that allows testing models on a vast lab of real physical devices to ensure accurate performance metrics.
0:07:36 - On-Device RAG SDK: A new SDK for Retrieval Augmented Generation (RAG) enables language models to access and query large local datasets, allowing them to answer questions with specific knowledge that isn't built into the model itself.
0:08:37 - On-Device Function Calling SDK: This SDK allows a language model to do more than just generate text; it can execute specific application functions or APIs, enabling powerful "tool use" capabilities directly on the device.
0:10:26 - Simplified Hardware Acceleration with Light RT: The new Light RT runtime drastically simplifies the process of using hardware acceleration, reducing the code required to enable the GPU delegate by up to 80%.
0:11:56 - Re-architected GPU Delegate (ML drift): The GPU delegate has been completely rebuilt as "ML drift," specifically optimized for generative AI models. It offers up to a 12x performance improvement for small language models compared to other on-device frameworks.
0:12:42 - Streamlined NPU Support: Light RT makes it simple to leverage NPUs (Neural Processing Units) on flagship phones, delivering performance gains of up to 25x over CPUs. The platform unifies the toolchains from partners like Qualcomm and MediaTek.
0:15:16 - Real-World Impact: The presentation highlights partner success stories, including Nsign Infocurity, which reduced its deepfake detection time from 67 seconds to 1.2 seconds, and Argmax, which saw a 30% performance gain for its speech-to-text SDK.