This video details the latest advancements in Google AI Edge, focusing on bringing Large Language Models (LLMs) to mobile and web devices. The presentation covers the expansion of supported models, new tools for model customization including fine-tuning and INT4 post-training quantization for significant size and speed improvements, and architectural modifications using the AI Edge Torch Generative API. It also introduces critical development tools like Model Explorer for visualization and AI Edge Portal for robust multi-device testing. Furthermore, two new powerful SDKs are unveiled: on-device Retrieval Augmented Generation (RAG) for enhancing LLM knowledge and on-device Function Calling for enabling LLMs to execute application actions. Infrastructure improvements through "Light RT" are highlighted, simplifying GPU acceleration (with a 12x performance boost via ML Drift) and leveraging MPUs for even greater efficiency (up to 25x vs CPUs). Real-world applications from partners like Nsign Infosecurity, Argmax, and Audio Shake demonstrate the practical impact of these on-device AI capabilities.
Bringing AI Edge to Mobile and Web: New Tools and Performance Enhancements
0:00 Introduction to AI Edge: Matias, Engineering Lead for Google's AI Edge team, introduces the latest advancements in bringing language models to mobile and web, emphasizing benefits like real-time processing, offline use, privacy, and reduced server-side costs.
1:57 LLM Inference API & Expanded Model Support: The LLM inference API provides a simple prompt-in, response-out interface for running small language models on the edge, supporting web, iOS, and Android. Google has expanded model support, partnering with Hugging Face for the "lightert" community, and now supports DeepSeek and Jamma models.
2:51 Model Customization - Fine-tuning: Mark discusses how to customize models. Developers can find edge-optimized Python implementations on Google AI Edge GitHub, pair them with original or fine-tuned weights, and convert them to an edge model format. Community tools like Torch Tune or Hugging Face Transformers can be used for "Laura" or full fine-tuning.
4:13 Model Customization - Quantization: The AI Edge Quantizer now supports INT4 post-training quantization, reducing model size by up to 8 times and significantly improving speed with minimal quality loss, optimized for language models.
5:00 Model Customization - Architectural Changes: The AI Edge Torch Generative API allows developers to make architectural changes or introduce new language models by combining PyTorch native layers with AI Edge Torch layers optimized for edge inference.
5:28 Inspectability & Testability - Model Explorer: Model Explorer is a graph visualization tool for intuitively viewing and debugging ML models, including hierarchical views and side-by-side comparisons of different model variants or quantization schemes. It has over 100,000 downloads.
6:16 Inspectability & Testability - AI Edge Portal: AI Edge Portal is a new GCP service (currently in private preview) that allows developers to run Light RT models across many real devices in Google's device lab for benchmarks, bulk inference, and evaluations, mirroring production environments.
7:34 On-device Retrieval Augmented Generation (RAG) SDK: Small language models often lack specific knowledge; RAG addresses this by parsing, storing, and retrieving relevant information during inference. The SDK includes an on-device embedding model, chunking method, vector database, and retrieval function, available on Android, with iOS and web support coming later.
8:37 On-device Function Calling (Tool Use) SDK: This SDK enables LLMs to take actions by executing registered application APIs instead of just returning text. It supports out-of-the-box use or customization via synthetic data generation and fine-tuning. Available on Android, with iOS and web versions later.
10:06 Infrastructure Improvements - Light RT: Megan discusses improvements to the developer experience with Light RT, simplifying GPU acceleration. Previously complex multi-step processes are reduced to two steps: loading and compiling the model with a specified backend.
11:56 GPU Acceleration with ML Drift: The GPU delegate has been rearchitected as "ML Drift" for even better performance, achieving a 12x improvement for small language models compared to other GPU-enabled inference frameworks.
12:40 MPU Acceleration: Leveraging specialized silicon (MPUs) in flagship phones offers significant advantages: 25x performance improvement over CPUs and at least 8x over GPUs in internal testing. MPUs provide lower latency, reduced processor, memory, and power utilization, enhancing device health and battery life.
14:18 MPU Vendor Partnerships: Google is partnering with Qualcomm and MediaTek, unifying the developer experience for MPU compilation and distribution. AI Packs, in cooperation with Google Play Store, configure correct model distribution for supported devices.
15:12 Real-world Use Case: Nsign Infosecurity: This Singapore-based cybersecurity firm used Light RT tools to reduce deep fake detection time from 67 seconds to 1.2 seconds on device, making the technology more accessible while reducing resource consumption.
16:10 Real-world Use Case: Argmax: Argmax, building enterprise-grade on-device inference SDKs like WhisperKit (speech-to-text) and SpeakerKit (speaker identification), realized a 30% performance improvement by switching to Light RT and ML Drift, benefiting from cross-platform support and exploring MPU expansion.
16:40 Real-world Use Case: Audio Shake: Audio Shake, specializing in separating audio components for applications like DJ remixing or voice isolation, leverages Light RT's performance for real-time use cases that were previously impossible, focusing on model quality.
17:15 Conclusion & Resources: The team expresses excitement for the future of on-device AI and directs viewers to Google/AI-Edge for documentation and further information, encouraging developers to reach out.