Overview of AMD NPUs: Capabilities, Documentation, and Usage

• 0:00:04 Introduction to AMD NPUs:
  • AMD NPUs are being integrated into CPUs by major vendors like Intel and Qualcomm.
  • Used by Windows Copilot and other applications.
  • Often compared to NPUs in edge devices from different vendors, but also used in larger devices.
• 0:00:48 Comparison with Other Platforms:
  • The presenter compares AMD's framework to Halio, noting similarities in logic and quantization approaches.
  • AMD is considered an edge device platform, similar to and compatible in pricing with Jetson.
• 0:01:17 Documentation and Resources:
  • AMD provides comprehensive documentation on NPU usage, quantization techniques, and code samples on GitHub.
  • Three main categories of documentation: general NPU usage, quantization, and GitHub samples.
• 0:01:38 Advantages of AMD Platform:
  • Good documentation.
  • Multiple quantization approaches (e.g., quantization-aware training, mixed-precision quantization).
  • Support for various inference frameworks (ONNX Runtime, PyTorch, Llama.cpp).
• 0:03:43 Disadvantages:
  • Quantization is mandatory for NPU usage.
  • Limited function support, though substantial.
  • Small community compared to other platforms.
  • Limited model zoo.
  • Higher price point compared to some edge device platforms.
  • Modern VMS may not be fully supported yet.
• 0:05:29 Model Export and Quantization:
  • Models can be exported to ONNX or PyTorch.
  • Quantization is a crucial step, with options for different quantization strategies.
• 0:06:12 Inference Process:
  • Initialization involves setting up the ONNX Runtime with specific provider options and environment variables.
  • Inference process is similar to standard ONNX Runtime usage after setup.
• 0:07:47 Performance Benchmarking:
  • NPU outperforms CPU for many networks.
  • Not all networks can be exported to the NPU due to quantization and other limitations.
• 0:08:30 Conclusion:
  • AMD offers a robust infrastructure for NPU usage, comparable to Jetson and Hailo.
  • Currently not the cheapest solution and has some limitations in network support.
  • AMD is actively developing and improving the platform.
• 0:09:27 Open Questions and Future Work:
  • Further testing needed for different quantization strategies and mixed-precision approaches.
  • Quantizing and evaluating the accuracy of llms on the platform is an important area for exploration.
  • The basic pipeline is functional and user-friendly.