This user guide, "i.MX Machine Learning User's Guide," details the NXP eIQ Machine Learning Software Development Environment (NXP eIQ), which provides a comprehensive suite of libraries and development tools for deploying machine learning applications on NXP i.MX microcontrollers and application processors. It covers the software stack, including supported inference engines like TensorFlow Lite (recently LiteRT), ONNX Runtime, PyTorch, and Apache TVM, across various i.MX platforms and their computing units (Cortex-A, GPU, NPU like Ethos-U and Neutron-S).
The document elaborates on hardware acceleration mechanisms, delegate implementations for various NPUs and GPUs, application development flows using CMake and Yocto SDK, model quantization techniques, and runtime optimization. It also introduces NNStreamer for building complex vision pipelines and provides examples for image classification, object detection, hand gesture detection, and face recognition. Key updates include expanded LLM model support, LiteRT v1.2.0, ONNX Runtime 1.22.0, NPU power management features, and performance tuning for different i.MX series, along with detailed API references and supported operation lists.
Summary:
Document Overview (UG10166 Document Information):
This user guide, updated on June 26, 2025, details the NXP eIQ Machine Learning Software Development Environment for NXP i.MX microcontrollers and application processors.
Key Takeaway: NXP eIQ provides libraries and tools for ML applications, integrated within the meta-imx/meta-ml Yocto layer.
It supports four inference engines: TensorFlow Lite (LiteRT), ONNX Runtime, PyTorch, and OpenCV.
ML applications can target various i.MX processors and leverage Cortex-A, GPU, and NPU compute units for multi-threaded and accelerated execution.
Key application domains supported include Vision, Voice, Sound, and Large Language Models (LLM).
TensorFlow Lite (LiteRT) (2 TensorFlow Lite):
Key Takeaway: TensorFlow Lite, now called LiteRT, is optimized for on-device ML inference with low latency and small binary size, featuring extensive hardware acceleration.
It supports hardware acceleration via VX Delegate (i.MX 8 series), Ethos-U Delegate (i.MX 93), Neutron Delegate (i.MX 9 series with Neutron-S), and GPU delegate (i.MX 95 Mali GPU).
Features include v2.18.0, multi-threaded computation with Arm Neon SIMD, C++ and Python APIs, and support for quantized models.
Delegates: Built-in kernels (using RUY for performance), XNNPACK Delegate (CPU-optimized, default for quantized models), VX Delegate, Ethos-U Delegate, Neutron Delegate, and GPU Delegate (OpenCL-based for i.MX 95).
Flex Delegate (2.6 Enabling TensorFlow Operators in TensorFlow Lite Runtime): Allows using unsupported TensorFlow operators within TensorFlow Lite runtime by converting them to "Flex operators." Requires building the library with specific configurations, including Docker for compilation.
Application Development (2.5 Application development): Supports CMake projects (recommended for dependency resolution and static/dynamic linking) and using precompiled libraries from the Yocto SDK.
Post-Training Quantization (2.9 Post training quantization): Describes methods using TensorFlow Lite Converter, emphasizing per-channel quantization and the need for representative datasets for full-integer quantization. MLIR-based conversion is default.
TensorFlow Lite for Microcontrollers (TFLM) (2.10 TensorFlow Lite for Microcontrollers): A lightweight version for low-power applications on Xtensa HiFi4 (i.MX 8ULP) and Arm Ethos-U (i.MX 93), using static memory allocation and optimized kernels.
Running Examples (2.7, 2.8): Demonstrates image classification (e.g., label_image) and benchmarking (benchmark_model) with various delegates and options for CPU, GPU, and NPU acceleration.
ONNX Runtime (3 ONNX Runtime):
Key Takeaway: An open-source inference engine for ONNX models, providing a unified API for ML model acceleration across deployment targets.
Features v1.22.0, multi-threaded execution with Arm Neon SIMD, experimental Neutron NPU Execution provider (i.MX 95 for LLM), VSINPU (i.MX 8 series NPU/GPU), C++ and Python APIs, ONNX 1.17, Opset 23, Arm KleidiAI integration, and MatMulNBits support for 4/8-bit quantized weights.
Software Stack (3.1 ONNX Runtime software stack): Includes Inference Session, Graph representation, and Execution Providers (CPU, Neutron, VSINPU).
ONNX Model Test (3.2 ONNX model test): Uses onnx_test_runner to run standard tests from the ONNX Model Zoo, supporting CPU and VsiNPU.
LLM Model Support (3.2.2 Running an LLM model): Supports quantized LLMs (8-bit and 4-bit) on CPU and Neutron NPU (i.MX 95) with examples using llm_chat_randominput.py.
Performance Test (3.3 ONNX performance test): Uses onnxruntime_perf_test for benchmarking models.
PyTorch (4 PyTorch):
Key Takeaway: A Python-based scientific computing package for deep learning projects.
Features v2.3.0 and Python 3 support. Currently, only CPU is supported, with explicit engine specification (torch.backends.quantized.engine = 'qnnpack') required for quantized models.
Installation via pip and an image classification example (pytorch_mobilenetv2.py) are provided.
TVM (Apache TVM) (5 TVM):
Key Takeaway: An open-source ML compiler framework for optimizing and running computations on diverse hardware backends.
Features v0.7.0, compiles deep learning models into deployable modules, and supports i.MX 8M Plus platforms with OpenVX.
Workflow (5.1 TVM software workflow): Transforms models into Relay IR, applies optimizations, and generates code with intrinsic calls for offloading to accelerators.
Examples include RPC verification (running on remote devices) and individual execution on the device, with host-side compilation.
LiteRT (Experimental) (6 LiteRT (Experimental)):
Key Takeaway: Google's high-performance runtime, formerly TensorFlow Lite, with active development focusing on LiteRT packages.
Migration from TensorFlow Lite to LiteRT involves replacing the PIP package from tflite-runtime to ai-edge-litert; APIs remain the same.
Currently, only Python API is supported for example development.
Examples demonstrate running image classification on i.MX 8, i.MX 93 (Ethos-U), and i.MX 9 (Neutron-S) with their respective delegates.
NN Execution on Hardware Accelerators (7 NN Execution on Hardware Accelerators):
i.MX 8 Series (7.1 Hardware acceleration on i.MX 8 Series):
NN accelerators: Neural Processing Unit (NPU) for 8/16-bit fixed point, and Graphics Processing Unit (GPU) for fixed point and half-precision floating point.
Interface: OpenVX v1.3 with NN Extensions.
Profiling (7.1.2): Enable via U-Boot args (galcore.showArgs=1, galcore.gpuProfiler=1) and environment flags (CNN_PERF, NN_EXT_SHOW_PERF, VIV_VX_DEBUG_LEVEL, VIV_VX_PROFILE).
Warmup Time (7.1.3): Initial model graph initialization takes longer; can be reduced by caching OpenVX graphs using VIV_VX_ENABLE_CACHE_GRAPH_BINARY and VIV_VX_CACHE_BINARY_GRAPH_DIR.
GPU/NPU Switching (7.1.4): Use USE_GPU_INFERENCE=1 for GPU, otherwise NPU (default).
Quantization (7.1.5): Both per-tensor and per-channel quantizations are supported, with NPU optimized for per-tensor.
Ethos-U on i.MX 93 (7.2 Hardware acceleration with Ethos-U on i.MX 93 platform):
Dedicated NPU (Ethos-U65) at 1 GHz (0.5 Tops) for ML inference, integrated with Cortex-M. Supports quantized CNNs, TFLite/TFLite-Micro.
Vela Tool (7.2.4): Compiles TFLite models for Ethos-U, generating optimized _vela.tflite models with custom "ethos-u" operators and performance estimates. Requires pre-quantized models.
Inference API (7.2.5): Provides C++ and Python APIs (ethosu.hpp, libethosu.so) for dispatching inferences to the Ethos-U Linux kernel driver, handling device, buffer, network, and inference object creation. Includes Interpreter class for simplified usage.
Memory Hierarchy (7.2.8): Utilizes TCM (code, stack), OCRAM (NPU intermediate data), and DRAM (tensorArena/model weights/bias/IFM/OFM). Specific memory modes (Dedicated_Sram, Sram_Only) are used with Vela compilation.
Profiling (7.2.10): PMU profiling with Ethos-U delegate options (enable_cycle_counter, pmu_event0, etc.) to capture detailed event IDs.
eIQ Neutron NPU on i.MX 9 Series (7.4 Hardware acceleration with eIQ Neutron NPU on i.MX 9 series platform):
NXP-developed Neutron-S NPU (Neutron computation core, RISC-V controller, Data Mover) to accelerate ML inference.
Supports quantized CNNs, TFLite inference with custom neutronGraph nodes.
Performance Tuning (7.4.3): Recommendations for i.MX 95 include disabling DDR clock gating, setting CPU to performance mode, and disabling CPU idle.
Power Management (7.4.4): Neutron device supports Automatic (default), Performance, and Low Power modes with configurable suspend delay.
Vision Pipeline with NNStreamer (8 Vision Pipeline with NNStreamer):
Key Takeaway: A flexible GStreamer-based framework for neural network applications, enhancing media pipelines with ML capabilities.
Adds other/tensor and other/tensors data types and provides various tensor_* plugins (converter, filter, decoder, transform, etc.) for ML operations.
Object Detection Example (8.1 Object detection pipeline example): Illustrates a pipeline on i.MX 8M Plus using v4l2src, imxvideoconvert_g2d, tensor_converter, tensor_filter (TensorFlow Lite with VX Delegate), tensor_decoder (bounding boxes), imxcompositor_g2d, and waylandsink.
Profiling (8.3 Pipeline profiling): NNShark (based on GstShark) monitors SoC hardware usage (GPU, NPU utilization, bandwidth) and can integrate with power measurement tools.
eIQ Demos (9 eIQ Demos):
Provides TensorFlow Lite demos for i.MX 93, requiring models to be downloaded.
Includes: Image Classification, SSD Object Detection, Hand Gesture Detection, and Face Recognition (all Python examples).
Release Notes (10 Release Notes):
Known Issues (10.1): i.MX 95 inline model compilation, i.MX 8 dynamic shapes/weights, and broken model download links.
LF6.12.20_2.0.0 Updates (10.2): Highlights Neutron converter optimization, LLM model support (ONNX Runtime CPU/EP and LiteRT CPU), LiteRT v1.2.0 (experimental), ONNX Runtime 1.22.0 (Neutron EP, VsiNPU EP, MatMulNBits, KleidiAI), Arm Vela Compiler 4.2.0 for i.MX 93, and NPU Power Management for i.MX 95.
API Reference (12 Neural Network API Reference): Lists supported Neural Network operations for TensorFlow Lite and ONNX Runtime.
OVXLIB Operation Support (13, 14): Detailed tables summarizing supported OVXLIB operations on NXP GPU and NPU hardware (NN, TP, PPU execution engines) across various tensor formats.