Deep Dive into INT8 Tensor Core Matmuls on Turing GPUs: Educational Presentation and Optimization Techniques
0:00-0:01 Introduction Eric Schul presents on INT8 tensor core matrix multiplications (matmuls) for Turing GPUs, emphasizing education over state-of-the-art speed.
0:01-0:04 Motivation for INT8 Matmuls Interesting problems not just on GPUs but across range of hardware. Why is it different from if you do floats?
0:04-0:07 ABX 512's vni extension Provides instructions that do four-way dot product interests.
0:07-0:11 Cuda's dp4a instruction Takes four 8bit integers of first operand and four 8bit integers of second operand to dot product between those four. Then accumulates into a 32bit accumulator which has same size as original input operand.
0:11-0:17 Advantage of Integer Multiplication The results are deterministic (unlike floating points). No need to figure out correct tolerances for test cases.
0:17-0:21 Goal of Eric's Demonstration Basic use of tensor cores directly in Cuda without large libraries or inline assembly.
0:21-0:24 Why inline assembly is sometimes still required The only way to use newer tenso instructions because it has no equivalent in C++. Code examples in Cutlass' source code also ends in inline assembly statement.
0:24-0:28 Purpose of function The main use is to give a simple start into using tensor course. The instructional value is the reason they haven't added anything new for hopper and newer instructions.
0:28-0:35 Code Characteristics Code focuses on nice, very large Square matrices with sizes divisible by a large power of two.
0:35-0:41 Optimization Target Optimizing for Turing, specifically a Quadro RTX 400 (tu104) GPU. Reasons include simplicity and availability for online submission system testing.
0:41-0:45 Online Submission System Users can submit code to the presented website, where it undergoes tests for correctness, sanitation, and benchmarking on a Quadro RTX 400.
0:45-0:48 Optimization difficulties It is hard to write a kernel that can run fast on multiple different kinds of GPU, but if willing to write hyper specialized code, the specific GPU can be optimized.
0:48-0:55 Specializing for GPUs Tail effect can happen if the number of output blocks is not divisible by streaming multiprocessors, where a few stragglers can affect the whole iteration. So a specific block size can be selected to optimize for this.
0:55-0:57 Matrix multiplication testing (Testing Implementation Correctness) Several method can be used: Small Matrices, Special Matrices (Multiply a Matrix with an identity matrix), and structured Matrices.
0:57-1:00 Computation Perspective Fast algorithm is not included in test because that would be spoiling a little bit of the exercise.
1:00-1:05 Reference Result Used What's called FS algorithm - If matrix c is the product of a and b then any Vector X multiplied by c is the same as if multiplied first by b and then a. This method is cheaper. Third option is trusted library qas.
1:05-1:13 Matrix Multiplication Decomposition Can be done bottom up or decomposed into smaller matrix products (knife decomposition). Can actually decompose it cleverly and end up with 7 products which gives you an algorithm that in in a Big O notation is actually faster.
1:13-1:15 Choice of Precision Precision can depend on generation and type of device implementation. Can use blocks of 4x4 with constant numbers.
1:15-1:18 Problem of Strassen algorithm It gives lots more addition matrix additions, which is not cheap on actual hardware.
1:18-1:21 Decomposing Matrix into smaller submatrices of fixed size (fixed by 16) After assuming that matrix dimensions are divisible by 16, think of matrix as de composed of individual blocks 60x60 numbers.
1:21-1:28 Cuda functions The amount of function that one needs is pretty limited. There are 2 different load functions, 1 store function, a function to fill with constant, and then the MMA sync.
1:28-1:31 Matrix Fragment The mapping of matrix elements into fragments internal storage is unspecified and subject to change in future architectures. Code only looking at nice matrices (large and square).
1:31-1:36 Implementation Code Writing it a way that parallels to tensor core becomes obvious.
1:36-1:40 Bad memory pattern issues Speed of light analysis needs to be done. In order to reach tensor performance, a factor needs to be added. Needs to put data in the right place and right time.
1:40-1:43 Memory Chart The graph shows that the memory reads that happened is far greater than what GPU has.
1:43-1:47 Warp Stalls The cycles for which instruction did not do anything useful, but if this increases, is really terrible for performance because it can't hide latency.
1:47-1:50 Col Major Format Second matrix should be in col major format. The transpose should be done.
1:50-1:53 O Notation Back to the big o notation, matrix is is n sq, but computation matrix computations products is n cubed, so the transpose will be really really cheap with the matrix.
1:53-1:56 Showed graph with results of optimizations and improvements Using col major got several x magnitude faster.
1:56-2:00 What is next optimization Next is to use this memory and right place on memory hierarchy. Graph shows the L1,L2 cache which is the main path.
2:00-2:07 Registers Reuse memory as much as possible at the fastest memory (registers). Then switch from W to paneling 3x3 fragments.
2:07-2:16 Nested Loops In the beginning of loop you can load 3 results that want to have fragment, and at the main computation, pre loaded Y fragments are used.
2:16-2:21 Next Graph, another 60% reduction or almost 70% reduction for memory access
2:21-2:26 Discussion, with help from Viam, about Mi(memory IO), Scoreboard, stall short scoreboard, and terminology
2:26-2:32 Vectorized Load INstructions The only load instructions has 60 bytes. Because there is not int 8 and 16 vector type, pretend that the int4 instructions that get memory alignment. The memory alignment is what allowed the compiler to choose the faster load instructions.
2:32-2:36 Next improvement: Pre-shuffling Pre-shuffling reads matrix once and writes it once. Copy of matrix in linearly addressable memory format.
2:36-2:43 Advantages of Pre-shuffling Also able to use MMA load direct to load and writing to shared memory.