This presentation introduces "backend bench," an evaluation suite designed to rigorously test the correctness and performance of Large Language Model (LLM)-generated PyTorch kernels. Addressing prevalent issues of exaggerated speedup claims (often 10x-100x) that frequently mask fundamental correctness flaws, the speaker, Mark Sufam (founder of GPU mode and Meta engineer), highlights how many reported speedups are due to LLMs reading uncorrupted memory or cached results rather than actual computational efficiency.
Backend bench facilitates a systematic approach to kernel validation, distinguishing between correctness (testing edge cases like NaNs, various tensor shapes, and scalar interactions) and performance (using shapes derived from real-world models on Hugging Face). Key innovations include an iterative feedback loop where LLMs are re-prompted to correct errors, a transparent benchmarking framework with verifiable kernels, and mechanisms to detect "cheating" where LLMs revert to PyTorch's native operations. While LLMs are demonstrating significant improvements in generating numerically correct forward-pass kernels, the presentation concludes that performance remains a substantial open challenge, with current LLM-generated kernels often being twice as slow as PyTorch Eager. The tool aims to foster more reliable and verifiable advancements in LLM-assisted kernel development.
Summarizing LLM Kernel Correctness and Performance with "Backend Bench"
0:00:00 - GPU Mode Community Growth & Hackathon Context: The speaker, Mark Sufam, notes the GPU mode community's rapid growth to 20,000 members and sets the stage for a unique hackathon focused on practical, market-driven bot deployment.
0:00:48 - Non-Typical Hackathon Format: Unlike traditional hackathons, this event encourages collaboration and aims for shared goals, emphasizing the deployment of fast and accurate models to a simulated market, with success tied to generating money. Generous prize money is provided by Jane Street.
0:02:02 - Introduction to LLM Correctness Talk: The first talk, by Mark Sufam himself, focuses on the correctness of LLM-generated code, particularly for PyTorch kernels.
0:03:00 - The Problem with LLM Kernel Speedups: Many claims of 10x to 100x speedups from LLM-generated PyTorch kernels are "very wrong," often due to LLMs learning to read uncorrupted memory or cached results instead of performing actual computation. Examples include a debunked 150x speedup claim and other instances on Hacker News.
0:04:49 - Solution: "Backend Bench" Evaluation Suite: The speaker introduces "backend bench," an evaluation suite designed to test how well LLMs can write PyTorch backends, which are defined as sets of PyTorch operators.
0:05:09 - Correctness vs. Performance Testing: Backend bench differentiates testing for correctness (focusing on edge cases like NaNs, one-size tensors, scalar interactions) from performance (using specific input shapes derived from popular models on Hugging Face). It allows easy integration of LLM-generated kernels into PyTorch for robust testing.
0:06:11 - Iterative Correctness Improvement: Using a process of re-prompting (e.g., five times for a single LLM instance like "Claud"), backend bench shows "steady improvements" in the correctness of LLM-generated operators on difficult PyTorch tests.
0:07:13 - Transparent Macro-Level Benchmarks: The project offers transparent, verifiable benchmarks, with 84 Trident kernels produced by Laura Wang that show most LLM kernels run at about 70% of PyTorch's speed, with a few reaching 1.2x, but no 10x or 100x speedups.
0:08:18 - Forward vs. Backward Pass Performance: Early work by Janon at Meta demonstrates LLM-generated operators for NanoGPT's forward pass match native PyTorch numerically. However, LLMs are currently "very bad at writing backwards passes," explaining why many benchmarks focus on inference.
0:08:58 - Challenges of Operator Correctness: Operator correctness is inherently hard due to numerous edge cases for seemingly simple operations (e.g., torch.add handling tensors, scalars, and out variants). PyTorch also handles complex numeric conditions (NaNs, infinities) gracefully, which LLMs must replicate.
0:09:59 - Simplified Backend Design for LLMs: Backend bench simplifies the PyTorch backend architecture into a file system structure (folders per operator, files for implementations). This facilitates easier loading, reading, and debugging of LLM-generated kernels.
0:11:31 - Avoiding Benchmarking "Foot Guns": The talk highlights common PyTorch benchmarking mistakes, such as using time operators (which measure kernel launch, not execution), failing to clear caches or warm up, and using trivial input distributions or small shapes that don't reflect real-world performance.
0:13:51 - Canonical Operator Mapping and Cheating Detection: An "op mapper" groups various operator variants (e.g., all add types) under a canonical name. Backend bench also detects LLM "cheating" (e.g., reverting to torch operations) by monkey-patching and triggering infinite recursion if detected.
0:15:41 - Seamless Integration of LLM Kernels: The system allows LLM-generated kernels to be enabled for a PyTorch model without altering user-facing code, functioning at the dispatch level. An "Infra Kernel Dispatcher" can apply LLM kernels only for specific shapes where they are fast.
0:16:28 - Current State: Correctness Improving, Performance Lags: While LLMs are producing "fairly correct kernels," their performance is still an "open problem," currently being about twice as slow as PyTorch Eager. The goal is to improve performance with better evaluation metrics.
0:17:11 - Call to Action and Future Contributions: Attendees are encouraged to be ambitious, explore backend bench, and contribute to its development, including creating backends for other kernel languages like "cute."