This video features two presentations from a bootcamp on GPU programming for AI.
The first talk, by William from Anthropic, focuses on building a correct mental model for GPU performance by moving beyond CUDA's software abstractions. He explains that a GPU is a collection of a fixed number of Streaming Multiprocessors (SMs), each containing smaller "warp schedulers." He reconciles the high number of software threads with the limited physical hardware by detailing three core concepts: SIMD execution where a single instruction operates on a "warp" of 32 threads, time-multiplexing of multiple warps to hide instruction latency, and the serial execution of thread blocks when their count exceeds the number of SMs. This hardware-centric view is used to explain common but counter-intuitive performance behaviors.
The second talk, by Siman from Stanford, addresses the challenge of writing and porting high-performance AI kernels. She introduces Thunderkittens, a programming framework designed to offer simple yet powerful abstractions. The core principles include using hardware-aligned "tiles" as the base data type to abstract away complex memory layouts, providing a PyTorch-like API for operations, and using a producer-consumer kernel template to manage asynchronous execution and resource allocation. This approach simplifies development, enabling the rapid creation of efficient kernels and the co-design of hardware-aware AI architectures, as demonstrated with the "Based Linear Attention" model.
GPU Programming Fundamentals and Abstractions for AI Kernels
0:00:47 GPU Programming: A Hardware-First Approach: William introduces the goal of the first talk: to understand GPU performance by focusing on the physical hardware rather than the CUDA software abstractions of threads and blocks.
0:05:39 Common Performance Puzzles: Two counter-intuitive scenarios are presented: 1) Why reducing threads per block (e.g., from 1024 to 256) doesn't always hurt performance, and 2) Why a tiny increase in thread blocks (e.g., from 128 to 133 on an H100) can cut performance in half.
0:09:04 The Real GPU Architecture: A GPU is physically composed of a fixed number of Streaming Multiprocessors (SMs), typically around 100-132. Each SM contains four smaller compute units called warp schedulers.
0:16:37 How GPUs Run So Many Threads: The discrepancy between hundreds of thousands of software threads and a few hundred hardware schedulers is explained by three key mechanisms:
SIMD Execution: Instructions are executed on "warps" (groups of 32 consecutive threads) simultaneously.
Time Multiplexing: A warp scheduler juggles multiple warps, issuing instructions from a ready warp while another is stalled (e.g., waiting for memory), which hides latency and keeps hardware busy.
Serial Block Execution: If the number of launched blocks exceeds the number of SMs, the SMs will execute them sequentially.
0:58:43 Solving the Performance Puzzles: The hardware model explains the puzzles: 1) 256 threads are often enough to fully saturate an SM's warp schedulers, so adding more provides no speedup. 2) On a 132-SM GPU, launching 133 blocks creates a "straggler" block that runs in a second wave after the first 132 are complete, nearly doubling the runtime while most of the GPU sits idle.
1:14:48 The Challenge of AI Kernel Development: In the second talk, Siman explains that writing efficient and portable AI kernels is extremely difficult, creating a gap between the theoretical efficiency of new AI models and their actual wall-clock speed. The primary rule is to "keep the tensor cores fed," as they provide the vast majority of the compute power.
1:48:33 Thunderkittens: Simple Abstractions for Peak Performance: Thunderkittens is introduced as a C++/CUDA-based DSL that simplifies kernel development with three main abstractions:
Tiles: A core data type that matches the hardware's native instruction shape (e.g., 16x16), abstracting away complex memory layout management.
Functions: A PyTorch-like library of operations (MMA, reduce, etc.) that operate on tiles.
Kernel Template: A producer-consumer template that manages the complex scheduling and synchronization of asynchronous work across warps.
2:05:51 Mega-kernels for End-to-End Performance: By removing artificial boundaries between kernels (e.g., attention, MLP, normalization), an entire model layer can be fused into a single "mega-kernel." This allows for more advanced scheduling and significantly improves performance over traditional frameworks like vLLM.
2:11:35 Co-Designing AI Models and Hardware Kernels: Using Thunderkittens, the "Based Linear Attention" architecture was developed. It was co-designed to be highly efficient on modern GPUs by minimizing slow memory movement and maximizing on-chip computation, effectively balancing model quality with hardware performance.
2:18:41 Future Directions in AI Systems: The talk concludes by highlighting the importance of building hardware-aware AI architectures and preparing for a "multi-silicon future" where understanding communication and memory across different hardware platforms (like Nvidia's NVL72 or AMD GPUs) will be critical.