Int8 Tensor Core Matmuls on Turing GPUs: An Educational Deep Dive
0:00-0:01 Introduction of Eric, an LMC core developer, who will discuss Int8 tensor core matmuls for Turing.
0:01-0:01 Goal: The presentation aims to educate, not showcase state-of-the-art speed. Focuses on basic implementation using CUDA.
0:01-0:02 Rationale: Int8 matrix multiplication presents interesting challenges across various hardware types (CPUs and GPUs) due to the size increase from 8-bit to 16-bit results.
0:02-0:03 AVX-512 and CUDA comparison: AVX-512's VNNI extension and CUDA's dp4a instruction offer specific instructions for integer dot products, requiring clever usage for signed matrix multiplications.
0:03-0:03 Tensor Core Modifiers: Explores using tensor cores, available since Turing GPUs, for Int8 matrix multiplication.
0:03-0:04 Benefits of Integer Modifiers: Integer matrix multiplication yields deterministic results, eliminating the need for complex tolerance calculations in testing.
0:04-0:04 Instructional Goals: Aims to demonstrate basic tensor core usage directly in CUDA (w/ QA C++); avoiding large libraries like CUTLASS or inline assembly.
0:05-0:06 Inline Assembly Question: When is inline assembly required versus using CUDA C++ APIs? Newer tensor instructions (Warpgroup MMA) often need inline assembly, while CUDA headers generate inline assembly for some functions, abstracting the complexity.
0:06-0:07 Assembly Lag and Specialization: C++ APIs have inherent lag, and for super deep specialization, assembly may be best. CUDA C++ APIs can provide more instruction if you do not want to specialize.
0:07-0:08 CUDA's approach and value: Cuda gives a simple start to Tensor Courses - and for any real specialisaiton one would need to use inline assembly.
0:08-0:08 Nice Matrix Constraints: For simplicity, the code focuses on large, square matrices with dimensions divisible by large powers of two, avoiding boundary condition issues.
0:08-0:08 Optimization Target: Code is optimized for Touring architecture specifically Quadro RTX 4000. Working on Touring is that it has fewer features to worry about + an available submission system uses this GPU.
0:09-0:10 Course testing available: An online submission system is available for testing Int8 tensor core implementations on a Quadro RTX 4000, including correctness, memory access and benchmarking tests.
0:10-0:13 Kernel effort for GPU specific performance?: There are different generation requirements for GPU kernels, and also fine tuning is often required. In Q C++ it can still work on both during + Amper though.
0:11-0:13 CUDA Kernel: The CUDA is compatible for both Amper + Touring even though assembly is a little different.
0:11-0:11 Block sizes: Tail effects can occur (some blocks running longer than others) that one would like to optimize
0:13-0:13 Debugging tests: Random testing vs using small or structured matrices is discussed for easier debugging and less computational overhead.
0:15-0:15 Checking whether alg is correct is important: But using navie is often very slow.
0:15-0:17 Use special algorithms: A Fast Algorithm approach with vector x is one that has high probability, high cost, easy to determine if correct, no QUDA necessary.
0:17-0:17 QUDA Approach for testing: Simple, easiest, but requires QUDA and downloading.
0:17-0:18 Matrix Multiplication: Matrix A has dimensions N * K, Matrix B has dimensions K * N. Output has N * N.
0:18-0:18 High Level Bottom up: In a large mtri you can decompose into a smaller Matrix Products
0:19-0:20 What precision to run operation?: Sten alg is rarely used in hardware as there are many Matrix additions.
0:20-0:21 Fix Matrix to a certain fixed size: And assume Matrix dim are divisible by 16 for no boundary conditions.
0:21-0:23 CU Face and functions: Only 2 different load functions, 1 store functions and Flll. Mma sync operation is for Multiplying 2 matrices.
0:23-0:24 Code Description: A is written to a way where the code parallels are obvious. Result fragment is 1 result.
0:24-0:24 CUDA: the loops needs to be written in steps of 16 or less. And all that really changes is the mma synch.
0:24-0:25 All code is written on slices and will be available: the presentation will go over where to try it out.
0:25-0:25 New code: It's simple to run faster right: wrong. Speed of light.
0:25-0:26 Get a Decent Um Performance: You need spec sheets and know to convert performance and understand the specs.
0:26-0:26 Data is important: Where is it? Putting Data at the right time is important so GPU does work.
0:27-0:28 Issue: Bad memory access pattern. And that loadMatrix function Fetches to Global memory distributes in threads.
0:28-0:30 B matrix should be collum major: And needs a transport kernel.
0:30-0:30 Benefit: the transpose is small because the kernel code has N-squared and matrix is N-cubed which is for a much better ratio.
0:31-0:32 Speed up vs slow kernel: 6 milliseconds vs 1.5 sec.
0:32-0:34 High memory graph transfer: A large percent to to L1 and also between other areas.
0:32-0:35 Cycles: number cycles spent in some state didn't do anything useful.
0:32-0:34 Stall throttle means: The War want to start looting new data from Global memory but limited wait.
0:34-0:34 Stall long scoreboard: The loaded instruction went through the request memory by then one use.
0:34-0:35 Both warps are important: But try to get rid of them. And the main goal is put mem in the right place.
0:35-0:36 Instructions: Decreased by 45 percent by high preformance.
0:36-0:36 Q optimization toolkit: The cused reuse same data makes you you some it to memory.
0:36-0:37 Memory: There is 2 versions one register is 45 and then shared men is the other.
0:37-0:37 Reduce register and improve throughput: Use as much memory in the right place. Use reg to read as well as use reuse
0:37-0:38 High Memory Bandwidth: the big O has not improved but they do have improved memory bandwidth.
0:38-0:38 Data Reuse: Thread co using to reuse as much memeory is possible and only can if each tread does more output do.
39:00-0:40 Threat: Tile code with memory access and nested loops do the accumulation.
0:40-0:40 Memory reduced on Ncu: the chart and all data is put.
0:40-0:42 Memory cycle: 1 side high cycle helps.
0:42-0:43 Noted what are drawbacks of this code: increasing code increases register.
0:43-0:43 Increase: code increases reg.
0:43-0:44 Another memory available: This should memory has some advantages.
044-0:44 Why 3: it reduces occupancy better than too and it is the best balance.
0:44-0:46 the code flows: but code must be tried on and error to know what will work best on benchMarking.
0:45-0:46 Register Reuse: can be a bottle neck.
0:46-0:47 Shared memory in high memory, so will it work.
0:47-0:47 the design: 9 walls and 3 by 3 tiles. In a GPU block and 3 level tile, fragment tile 16. So basically 144 by 144.
0:47-0:48 Data Load: Can now load this vector with linear load for it.
0:48-0:49 Reinter code trick: Can rewrite as in4 is the same data that there force needs it so it can be done fast.
0:49-0:49 Complication: Tells compiler is in4 data with 16 byte boundary and that allows fast code.
0:49-0:50 Transfer Verbatim Copy: from Global memeory share using Vector.
0:49-0:50 New Gpu could do more fun stuff: Like M memory copy. This version is not good but on too during that not do the code is what this test is.
0:50-0:50 Sizing: It gives you an amazing tool that can solve you, it's time to grow.
0:51-0:53 code memory and data: With the new memory codes the data with the code is more important for cycles.
0:53-0:53 Another Test code: For a performance code.
0:53-0:55 Ncu's: code increases the memory
0:55-0:56 A large pre-process test: In this slide and high light was added.
0:56-0:56 Long Code: long cycles are done in the code.
0:56-0:56 Helper functions: is more needed for high level code but compiler does wierd Knobs.
0:57-0:58 New speed code: It needs no and not.
0:58-0:59 Help with Data: But Data it's all.
0:59-0:59 This Presentation is only: 40 percent correct.
0:59-1:00 New Code: all that has been added for the code was used.
1:00-1:01 To reduce cycles: It's now all better to show code.
1:00-1:01 Miyo codes: It's what high is and low, it's cycles and needs more more
1:01-1:02 cycles or more data: If high there are many problems.
1:02-1:04 The memory needs more in it: It can be shown.
1:04-1:05 All New Speed Code: All code can do the high level easy in all great.
1:05-1:06 In the and side what this does and code all are done with no code is low: so all can be made to be high
1:06-1:07 Long code and long is in high: so it's now great
1:07-0-10 New code are done and can code was high and high codes can be code.
1:10-1:11 All the all code data all code: was bad and that that is were what is needs
1:11-1:12 It can change and all the data does. But the long long with data are all the bads.
1:12-1:12 This code is over and all you need to see and do is come can make this a grade data.
1:12-1:13 What should be the next code data: to put in the can code and data.
1:13-1:14 What will help the code to become right and what do not.
1:14-1:15 A Data test is can be good for any all.
1:15-1:16 All it can take to become right so what should be on high data what do I now high low to code.
1:16-1:17 If code can code right high the high be the code with high are on good and great.
1:17-1:17 If data low all that what be high high and see if all are good.
Future optimizations in slides will follow this list