Int8 Tensor Core Matmuls for Turing: A Deep Dive into Optimization and Cuda Programming
00:00:21 Introduction: Eric Schul, an LMC core developer, presents an educational talk on int8 tensor core matmuls for Turing GPUs.
00:01:06 Goal and Context: The presentation focuses on the basics of int8 matrix multiplication and tensor cores, not state-of-the-art performance. Int8 matmul is an interesting problem across various hardware.
00:01:36 Why Int8? Int8 matrix multiplication presents unique challenges on CPUs due to register size differences. AVX-512 and CUDA have specific instructions like VNNI and DP4A for int8 operations.
00:03:30 Advantages of Int8: Integer calculations are deterministic, eliminating tolerance issues in testing.
00:04:40 Direct CUDA Approach: Demonstrates using tensor cores directly in CUDA without external libraries or inline assembly.
00:05:09 Inline Assembly: Inline assembly is necessary for newer tensor instructions without C++ equivalents (e.g., Warp Group MMA). Cuda C++ APIs has inherent lag.
00:06:46 Instructional Value: The speaker believes the primary value of in-built functions from Cuda C++ is to give a simple start into using tensor cores.
00:07:17 Matrix Characteristics: The code uses large, square matrices divisible by a large power of 2 to avoid boundary issues.
00:08:16 Turing Optimization: Optimizations target a Quadro RTX 4000 (Turing TU104). Turing offers fewer features, simplifying optimization and an online submission system available for anyone to try out is being mentioned.
00:09:10 Online Submission and Testing: An online system allows users to submit code, run tests for correctness and performance, and receive a benchmark report.
00:10:00 GPU-Specific Kernel Optimization: Differences in GPU generations, especially compute capability, require specific kernel optimizations.
00:10:53 Compiler Influence: Compiler reordering affects instruction latencies. Specialization helps, such as selecting block size to avoid tail effects.
00:11:19 QC++ vs Assembly: Writing in QC++ ensures code works for Turing and Ampere.
00:13:13 Testing Methodology: Simple testing is insufficient, so use small matrices, identity matrices, or structured matrices to diagnose errors.
00:15:42 Fuzzing: Fuzzing and other methods allow for more rapid testing.
00:17:17 Naive Int8 Matmul: The basic matrix multiplication involves two 8-bit matrices resulting in a 32-bit matrix. Two decomposition methods exist: Bottom up (embarrassingly parallel) and top-down using several smaller matrices.
00:19:21 Optimizing Precision: The type and generation of the device dictate precision choices.
00:20:52 Fixed Size Submatrices: Matrices are decomposed into fixed size submatrices (16x16) using a naive algorithm with fewer iterations.
00:22:22 Required Functions: Four major functions: Load, Store, Fill, and MMA Sync for multiplying two Matrix fragments.
00:22:58 Unspecified Mapping: The C++ API may map Matrix elements differently between architectures.
00:23:41 Scalar Multiplication: Scalar matrix multiplication is used to illustrate parallels with tensor cores.
00:26:11 Speed of Light Analysis: A speed of light analysis helps show how far away from supposed performance they are.
00:26:45 Data Location: Data placement is crucial; the presentation emphasizes putting data in the right places at the right times.
00:27:21 Bad Memory Access: Load Matrix Sync fetches data from Global memory, which is a bad memory access pattern.
00:27:54 B Row Major With B in row major the access is stride.
00:28:17 Memory Graph Analysis: Examining the memory graph reveals excessive data transfer between L2 cache and L1 cache.
00:29:02 Warp Stall Cycles: Analysis of warp stall cycles highlights the reasons for performance bottlenecks. Warp stall cycles are the cycles for which the war actually stalls.
00:29:17 Stall AGG Throttle: High due to limited amount of data one can wait in GPU to load new data from Global Memory. The load instruction had to wait for new data.
00:30:21 Scoreboard Stall Means the request for the data went through but the data wasn't there when they tried to use it.
00:31:06 Solution Col Major: Having A and B in row major is bad, and a major issue that should be solved, so setting B to col major gives linear access. You can set them by transposing them to that format in an additional kernel.
00:31:50 Square Time and Memory Costs N Square versus N Cubed so memory cheap.
00:32:27 Performance Gain A good speed up in perf, and they are doing more work.
00:32:58 Results: 60% Reduction in Memory, the code goes on, less addressing is needed.
00:33:17 WSS paradox: However WS stall is actually higher and we are doing less work at a higher rate.
00:34:10 Register spillage: Some are running for longer idle and do a full iteration till the end, also there is tail effect running in NVCU and with the limited streaming multiprocessor some run longer to the end, others don't.