This video features two talks on advanced GPU programming for AI. William from Anthropic provides a foundational understanding of GPU architecture, demystifying CUDA's software abstractions by explaining the underlying hardware of Streaming Multiprocessors (SMs) and warp schedulers. He elaborates on SIMD execution, time multiplexing, and serial block execution to explain common performance puzzles in CUDA kernels, emphasizing the importance of a hardware-centric mental model for optimization.
The second talk, by Siman from Stanford, focuses on "Thunderkittens," a DSL designed for hardware-efficient AI kernel programming. Siman highlights the immense cost of AI compute and the challenge of porting efficient kernels across hardware generations. She details Thunderkittens' abstractions, including tile-based data structures with optimized memory layouts, PyTorch/NumPy-like operations, and flexible kernel templates for asynchronous execution (e.g., producer-consumer). The framework aims to simplify complex GPU programming while achieving peak performance, enabling the co-design of AI model architectures with hardware capabilities, as demonstrated with "Based Linear Attention." Both talks underscore the need for programmers to understand GPU hardware for effective AI acceleration.
Exploring GPU Architecture and Hardware-Efficient AI Kernels
0:00 Welcome and Speaker Introduction: William, a performance engineer at Anthropic and former MIT PhD student, is introduced as the first speaker, focusing on foundational GPU programming.
0:47 Building a Strong GPU Programming Foundation: William's talk aims to provide fundamental understanding for anyone writing GPU kernels, applicable across languages like CUDA, Triton, or Cutlass 4.
0:04:06 The CUDA Programming Model: CUDA presents a software abstraction of blocks and threads. William challenges this, suggesting a deeper understanding of hardware is necessary for performance.
Puzzle 1: Why does reducing threads per block (e.g., from 1024 to 256) often not impact kernel speed?
Puzzle 2: Why can a small increase in thread blocks (e.g., 128 to 133 on an H100) drastically slow down a kernel?
0:08:50 GPU Hardware Overview: A GPU chip comprises many "tiny squares" called Streaming Multiprocessors (SMs) (e.g., H100 has 132 SMs), each containing four "warp schedulers" (akin to specialized CPU cores).
0:16:36 Bridging Software Threads and Hardware Schedulers: Thousands of software threads run on hundreds of hardware warp schedulers through three key mechanisms:
SIMD (Single Instruction, Multiple Data): Each instruction on a warp scheduler operates on 32 threads simultaneously (a "warp").
Time Multiplexing: Warp schedulers can juggle multiple warps (up to ~8) to hide instruction latency stalls.
Serial Block Execution: If too many blocks are launched for the available SMs, blocks are executed serially.
Threads & Blocks: Software abstractions for logical organization; threads within a block can communicate efficiently via shared memory. All threads in a block must run on the same SM.
Warps: Hardware-defined group of 32 consecutive threads that execute instructions in lock-step.
0:23:01 Thread Divergence: If threads within a warp execute different code paths (e.g., if-else), the hardware serializes these paths, impacting performance.
0:57:04 Key Numbers for GPU Utilization: An SM needs at least 128 threads to stay busy. A full H100 GPU needs around 17,000 threads for theoretical saturation.
0:58:43 Solving the Performance Puzzles with Hardware Knowledge:
Puzzle 1 Solution: Both 1024 and 256 threads per block can provide enough warps (at least 128 threads/SM) to saturate SMs, hence similar performance.
Puzzle 2 Solution: Increasing blocks from 128 to 133 on a 132-SM H100 causes the 133rd block to become a "straggler," running serially and monopolizing the GPU until it completes, drastically reducing throughput.
1:14:11 Siman's Talk: Hardware-Efficient AI Kernel Programming: Introduction to "Thunderkittens," a DSL for making AI kernels efficient across diverse and evolving hardware.
1:20:19 AI Hardware Overview (Hopper GPU): Highlights the heterogeneous nature of SMs and the critical role of Tensor Cores (16x faster than other compute in BF16), emphasizing the need to keep them busy.
1:25:05 Challenges in AI Kernel Development: Large tensors, complex data tiling, managing memory hierarchy, and optimizing non-Tensor Core operations (normalizations, reductions) are key difficulties.
1:26:17 GPU Parallelism Levels & Asynchrony: Kernel development involves managing parallelism at warp, multi-warp/SM, block, and grid levels. Modern GPUs offer many asynchronous mechanisms (loads, stores, matrix multiplies) and "wave specialization" (producer-consumer patterns) to maximize throughput and hide latencies.
1:37:20 The Gap Between Theory and Practice: Efficient AI architectures on paper often don't translate to wall-clock speed due to hardware incompatibility. Flash Attention, for example, required a complete rewrite for H100 for peak performance.
1:40:01 Thunderkittens: A New Abstraction Approach: Thunderkittens is a DSL offering concise, transparent programming abstractions to bridge the gap between theoretical and practical efficiency for AI researchers. It aims for simplicity and speed.
Tile-based Data: Uses a 16x16 tile as the fundamental data type, abstracting complex thread ownership and memory layout details, with layouts auto-optimized for Tensor Cores.
NumPy/PyTorch-like Operations: Provides familiar functions for operations on tiles.
HIP/CUDA Foundation: Written in HIP/CUDA, allowing graceful fallback and custom inline functions for new hardware.
Kernel Templates: Offers a producer-consumer template for asynchronous execution, simplifying warp coordination and synchronization, with tunable parameters.
Persistent Kernels: Uses a persistent grid approach for dynamic work pulling and control over block launch order for cache reuse.
Mega Kernels: Work on fusing entire AI models into single kernels to eliminate artificial boundaries and improve scheduling flexibility (e.g., Llama 1B/70B).
2:06:43 Thunderkittens Evaluation & Impact: Achieves peak performance across various AI workloads and hardware platforms with significantly fewer lines of code compared to complex C++/Cutlass solutions. Inspired other Pythonic DSLs.
2:11:18 Designing Architectures from the Hardware Up: Thunderkittens enabled the co-design of "Based Linear Attention," an architecture tailored for Hopper/Blackwell GPUs to balance quality and efficiency, demonstrating significant performance gains (5x utilization on Hopper++) over prior approaches.
2:21:50 Future Directions: Continued focus on bridging theoretical and hardware efficiency, and addressing challenges in the multi-silicon future, particularly multi-GPU communication and architectural design for emerging hardware.
2:26:05 Q&A Highlights: Discussions included compiler capabilities (CUDA graphs, loop unrolling), explicit warp-to-SM mapping (limited control), GPU context switching (none like CPUs), and the ongoing evolution of GPU programming frameworks (Mojo, Cute DSL). William and Siman believe GPU programming will get easier with converging hardware trends and better frameworks.
2:38:27 Session Conclusion: The talks conclude, offering contact information for further questions.