This video explores the paradigm shift from cloud-based to on-device AI, highlighting its transformative impact on privacy, speed, and reliability in everyday devices. It details the technical workflow for deploying AI models locally, utilizing Qualcomm's AI Hub for graph capture, compilation, validation, and performance profiling. A significant portion is dedicated to model quantization techniques, demonstrating how they reduce model size by up to four times and accelerate inference speed by a similar factor, crucial for efficient on-device execution. The discussion covers various runtimes (TensorFlow Lite, ONNX Runtime, Qualcomm AI Engine) and compute units (CPU, GPU, NPU), culminating in a real-time segmentation demo showcasing the dramatic performance gains of NPU-powered on-device AI.
On-Device AI: The Future of Intelligent Computing
0:08 The On-Device AI Revolution: AI is shifting from remote cloud servers to local processing on devices like phones, watches, and cars, promising instant, secure, and always-available intelligence.
1:01 What is On-Device AI?: AI models and algorithms run directly on the device, processing data locally without sending it to the cloud, giving devices their "own brain."
2:01 Key Advantages of On-Device AI: It offers total privacy (data never leaves the device), instant speed (no network latency), and offline functionality, making it superior for immediate, personal tasks compared to cloud AI's raw processing power.
3:01 Aura Product Concept: An example of on-device AI, Aura, is a screenless, proactive wellness companion designed to sense stress from bio-signals, send silent vibrations, and process/store all sensitive health data only on the device, ensuring verifiable technical privacy.
4:51 Massive Market Growth: The on-device AI market is projected to reach over $36 billion by 2030, quadrupling in six years with a compound annual growth rate of nearly 28%, indicating a major industry shift.
5:44 Addressing the AI Gap: On-device AI acts as a bridge for the "AI gap"—the disconnect between industry excitement and public confusion/worry about privacy—by delivering tangible privacy and reliability.
6:50 Ubiquitous Applications: On-device AI powers various daily applications, including advanced driver assistance, keyboard language models, smart speakers, robotics, drone scanning, and photo editing.
7:33 Key Use Cases by Modality: Applicable across audio (text-to-speech, speech recognition), image/video (photo classification, QR code detection), and sensors (physical activity detection, handwriting recognition), including multimodal capabilities.
8:27 Why Run Models On-Device?: Four main reasons: cost-effectiveness (utilizing local resources), efficiency (faster local processing), privacy (data stays on device), and personalization (customized experiences without external data).
9:15 Deploying Models with Qualcomm AI Hub: A 5-minute process involving four steps: capturing the model as a computational graph, compiling it for a target device, validating its numerics on-device, and measuring its performance.
11:51 Graph Capture: Converting a neural network's code representation (e.g., PyTorch) into a portable graph format using tools like PyTorch JIT trace, which captures computations for device deployment.
14:02 Model Compilation: Using the Qualcomm AI Hub's submit_compile_job API to optimize the traced model for specific physical devices (e.g., Samsung Galaxy S23) available in the cloud, generating a deployable artifact.
15:35 On-Device Runtimes: Popular runtimes include TensorFlow Lite (Android, mobile), ONNX Runtime (Windows), and Qualcomm AI Engine (Qualcomm embedded hardware). TensorFlow Lite is highlighted for its efficiency, flexibility, energy efficiency, and NPU compatibility.
16:50 Compute Units on Modern Devices: Models can run on the CPU (flexible, general-purpose), GPU (high-performance parallel computation), or NPU (extremely efficient for neural networks, up to 10x more efficient than CPU on modern Qualcomm devices).
18:35 Performance Profiling: The submit_profile_job API runs the compiled model on a chosen device (e.g., Samsung Galaxy S23) and downloads a performance profile, showing metrics like inference time and memory consumption (e.g., 27.9ms on CPU).
20:18 Numerical Accuracy Validation: Comparing inference results between the cloud (PyTorch) and the on-device NPU using metrics like Peak Signal-to-Noise Ratio (PSNR); a PSNR > 30 dB indicates good accuracy, ensuring cloud and device produce similar results.
23:42 Model Quantization Benefits: Reduces model size (better storage), enables faster processing (fewer computations), and lowers power consumption (crucial for battery-operated devices), making models up to 4x smaller and faster.
24:07 Quantization Concept: Converting floating-point tensors (e.g., 32-bit per value) into integer representations (e.g., 8-bit per value) using scale and zero-point parameters, minimizing "quantization error."
Post-Training Quantization (PTQ): Applies quantization after training using calibration data to minimize error (requires a few hundred samples).
Quantization Aware Training (QAT): Incorporates integer values into the training process, often yielding more accurate models for challenging scenarios.
29:54 Post-Training Quantization Workflow Demo: Using the FFNET 40S model, calibration data (100 urban scene images), and the quantize_export function, the video demonstrates how to quantize a model, validate its accuracy, and profile its performance.
34:41 Quantization Impact: A demo shows the FFNET 40S model (originally 55 MB, 16.9ms latency) reducing to 13 MB and achieving 4.6ms latency (3.7x speed-up) on a Samsung Galaxy S24 after quantization, running entirely on the NPU.
36:05 Device Integration Pipeline for Real-time Applications: The process for integrating models into mobile applications, including camera stream handling, GPU-based pre-processing (YUV to RGB, downsampling), NPU inference, GPU-based post-processing (upsampling, smoothing, blurring), and proper packaging of runtime dependencies (TensorFlow Lite, GPU/NPU delegates).
39:46 Real-time Segmentation Demo: A live demonstration highlights a 30 frames-per-second real-time segmentation model. On the CPU, it ran at only ~1 fps (800ms per frame), but on the NPU, it achieved a smooth 30 fps, showcasing the dramatic performance benefits of NPU acceleration.