This presentation explores the landscape of Domain Specific Languages (DSLs) for GPU kernels, emphasizing the critical balance between developer productivity and hardware efficiency in the context of deep learning's scaling laws. The speaker, Tri, highlights three key DSLs: PyTorch (as a high-level abstraction), Triton, and the lower-level QDSL. Through examples like Softmax, Matrix Multiply (GEMM), GEMM+SwiGLU, and Attention, the talk demonstrates how different DSLs expose varying levels of GPU hardware hierarchy, directly impacting performance. While PyTorch offers maximal productivity with respectable performance, QDSL, despite its verbosity and steeper learning curve, enables finer-grained control over hardware features (e.g., warp/cluster reduction, asynchronous memory operations), leading to significant performance gains (15-30% on Softmax, 7-15% on fused GEMM+SwiGLU, 15-20% on Attention) for compute-bound kernels, often matching or exceeding highly optimized vendor libraries. The presentation concludes with practical recommendations for choosing a DSL based on the kernel type and the desired trade-off between development time and peak hardware utilization.
Summarizing Domain Specific Languages for GPU Kernels
0:01:12 The Need for DSLs: The speaker emphasizes the growing importance of Domain Specific Languages (DSLs) for GPU kernels, driven by the need to balance researcher productivity with hardware efficiency to keep up with AI model scaling laws.
0:02:28 Balancing Productivity and Efficiency: Maximizing "intelligence per dollar" requires both algorithmic/data efficiency and hardware efficiency. DSLs help achieve this balance by making researchers productive while ensuring code runs efficiently on hardware.
0:03:49 PyTorch as a High-Level DSL: PyTorch is presented as the highest-level DSL for machine learning researchers, facilitating algorithm expression and model building. PyTorch 2.x, with torch.compile, can automatically generate and fuse GPU kernels for significant speedups, making it the recommended first step for optimization.
0:04:27 Triton for Mid-Level Control: Triton is highlighted for its success in abstracting away complex CUDA details (e.g., memory coalescing, shared memory management, scheduling), lowering the bar for GPU programming and boosting productivity compared to raw CUDA.
0:05:02 QDSL for Low-Level Performance: QDSL is introduced as a lower-level DSL, offering power comparable to PTX (Nvidia's assembly-like language) but embedded in Python, providing fine-grained control over hardware.
0:05:17 Hardware Hierarchy Exposure: DSLs differ in how much of the GPU's memory and thread hierarchy they expose (threads, registers, thread blocks, shared memory, thread block clusters, grid). Triton simplifies by exposing only thread block and grid, while QDSL exposes all levels.
0:07:28 Softmax Example: torch.compile Simplicity: Implementing softmax with torch.compile is extremely simple (one line decorator), often yielding good speedups and kernel fusion automatically.
0:08:20 Softmax Example: Triton Conciseness: Triton allows expressing low-level softmax operations concisely (6-15 lines), offering more control than torch.compile while remaining productive.
0:09:34 Softmax Example: QDSL Detail: QDSL code for softmax is significantly more verbose (around 50 lines) because it requires explicit management of memory access (vectorization, asynchronous copies) and all four levels of reduction hierarchy (thread, warp, block, cluster).
0:12:38 Softmax Performance Gains with QDSL: On an H100 GPU, QDSL shows a 20-30% performance advantage for small softmax shapes (due to explicit warp reduction) and about 15% for large inputs (due to cluster reduction) compared to Triton/PyTorch. This demonstrates the performance trade-off for increased code complexity and hardware understanding.
0:14:44 Matrix Multiply (GEMM) Performance: For standard matrix multiplication on H100 (Hopper), QDSL and cuBLAS (CUDA C++ library) achieve similar near-peak hardware performance (around 800 TFLOPS, or 80% of max). QDSL can even slightly outperform cuBLAS for smaller matrices by implementing "ping pong epilogue" overlap.
0:17:41 GEMM + Epilogue Fusion (e.g., SwiGLU): For common operations like GEMM followed by a SwiGLU activation (frequently used in Transformers), QDSL, by enabling full kernel fusion in a single kernel, achieves a 7-15% speedup over torch.compile (which would use separate cuBLAS + Triton kernels).
0:19:11 Attention Performance on Blackwell: Early results show a QDSL implementation of Flash Attention on Blackwell achieving a 15-20% speedup over cuDNN, a closed-source, lower-level library. This highlights the potential of QDSL, especially when combined with algorithmic improvements.
0:20:00 The DSL Spectrum and Trade-offs: DSLs exist on a spectrum balancing productivity and performance. PyTorch (highest productivity), Triton (good balance), QDSL (more control, more performance), and raw CUDA/PTX (most performance, least productive).