Understanding and Optimizing CUDA Kernels with NVIDIA's Insight Tools
0:00 Introduction: The session focuses on optimizing CUDA kernels, addressing common questions about performance metrics and profiling techniques.
0:01 Guest Speakers: Magnus and Jackson from the Nvidia profiling team share their expertise. Magnus has 15 years of experience in profiling tools.
0:02 Talk Format: Magnus plans to spend 15 minutes on slides and then dive into profiling traces, offering a look at how a senior engineer analyzes kernel performance.
0:03 Focus on Interaction: Attendees are encouraged to ask questions, making the session interactive.
0:04 Overview of Developer Tools: Jackson introduces the CUDA developer tools suite, including debuggers, profilers (Nsight Systems and Nsight Compute), Compute Sanitizer, and IDE integrations.
0:06 Profiling Workflow: Nvidia's recommended workflow starts with Nsight Systems for a high-level overview of platform performance, followed by Nsight Compute for in-depth kernel analysis.
0:08 Nsight Systems: Provides a timeline-based view of CPU, GPU, memory, and networking activity.
0:09 GPU Metrics in Nsight Systems: Includes device-wide metrics like GPU utilization, instruction issue rate, tensor core activity, and PCIe/NVLink metrics.
0:12 Nsight Compute: The primary focus of the session, used for low-level kernel profiling to understand how kernels execute on the GPU hardware.
0:12 What is Nsight Compute? a prime tool for CUDA workload profiling, focusing on short-term data collection.
0:14 Instrumentation Options: The tool does not require code modification, but MVTX instrumentation and the -lineinfo flag in nvcc can enhance profiling capabilities (Source correlation)
0:17 Addressing Questions: The speakers answer questions about trade-offs in sampling-based profilers and the impact of interception on end-to-end latency, explaining that while there's overhead in the tool itself, the main concern is to not impact the kernel execution
0:21 Profiling collection: It can replay a kel multiple times, collecting different data at each pass. It takes time to save/restore state and lock clocks as it clears the caches. This happens under the hood and is transparent to the application
0:37 Cheat Sheet: a quick sheet listing some flags, tools, and filter commands one can use. It is important to specify the CUDA Kel to be profiled to filter out unnecessary ones.
0:39 Live Demo (Simple Vector Addition): Magnus starts a live demonstration of the tool, beginning with a simple vector addition kernel.
0:40 Summary Metrics: The tool's summary page presents key metrics like duration, throughput, utilization percentages for compute resources and memory.
0:40 Speed of Light and Bottleneck Analysis: The speed of light breakdown (compute vs. memory) helps identify the primary bottleneck in the kernel.
0:41 Rule Output and Recommendations: The system provides rules and recommendations based on the profiling data, linking to relevant sections of the report and suggesting potential optimizations.
0:41 Roofline Analysis: Provides an alternate view of performance, showing arithmetic intensity vs. achieved FLOPS, helping to determine if the kernel is memory-bound or compute-bound.
0:43 Memory Workload Analysis: Provides a detailed chart of the memory hierarchy, showing traffic between different cache levels and highlighting bottlenecks.
0:53 Warp Scheduler Statistics: Show the activity of warp schedulers, identifying stalled warps and their stall reasons.
0:56 Source Page: Correlates metrics with source code lines, allowing developers to pinpoint performance bottlenecks to specific lines of code.
0:58 Demonstrating Code Correlation: the code has low-level assembly instructions that are correlated through line info. the code explains there are lots of stalls and then jumps to another portion of the code
1:16 Python Example (Image Manipulation): A python example of an image is shared
1:19 Source annotations: The code explains there are global instructions.
1:20 Fixing Code and Diffing Reports: Shows a work flow to make reports for fixing parts of the codes.
1:27 Demonstrating the tools: The speaker shows how the process goes and and make a new profile then make comparisons and do baselines
1:27 Demonstrating Diffing of Reports: Magnus demonstrates how to compare reports to assess the impact of code changes, showing speed-up percentages and highlighting performance improvements.
1:28 Demonstrate a different version of a Kernel: Another report version shows that different parts of the code can use optimizations
1:28 Show Source Code: The code shows the source and explains the different metrics
1:29 Demonstrate how to load and test the Kernel: The code shows what the kernel is executing
1:31 Tensor Core Example: Another Kernel is shown in the code.
1:31 Demonstrate and read different portions of code: Another example is shared with the tools in mind to display important functions of the tools
1:31 Demonstrating and testing new Kernel: Shows to be able to perform the steps with the new kernel.
1:38 Python Report Interface: They have a Python module that allows the user to create more specific reports.
1:38 Closing: There is a question of why they have used a 3070 for a sample and that is because it is all what Magnus had on his local machine.
1:38 Other resources: the code shows links to various other resources that the user might use