Jim Keller, a veteran in chip design, delivers a keynote on the evolving landscape of Artificial Intelligence, drawing insights from his experiences at Tesla, Intel, and Tenstorrent. He argues that building AI necessitates a radical transformation in hardware-software interaction, moving beyond traditional scalar computing to complex, multi-dimensional tensor operations. Keller introduces the "pyramid of operation rates," highlighting the hierarchical nature of AI computation from high-level PyTorch programs to low-level silicon operations, emphasizing the need for robust abstraction layers. He champions an open-source strategy for AI hardware and software, citing its benefits in fostering innovation, attracting talent, and creating shared value. Critically, he addresses the economic challenge of current high-cost AI solutions, proposing a scalable, cost-effective approach through modular chip designs, efficient networking (Ethernet), and an "array of arrays" architecture inspired by the human brain's organization. The talk concludes with reflections on common pitfalls ("tricks") versus effective engineering principles, asserting that "solid semiconductor execution" and a bottom-up, layered approach are key to successful AI system development.
Revolutionizing AI Hardware: Jim Keller's Vision for Cost-Effective, Open-Source Computing
0:00 Introduction & Personal Reflections: Jim Keller, with a background spanning electromagnetics to computer architecture at companies like Intel and Tesla, kicks off his keynote, reflecting on the rapid pace of technology and personal anecdotes.
2:31 The Radical Transformation of AI: Keller emphasizes that building AI is a "radical transformation," not just running programs faster, and acknowledges his own past "mistakes" in initially underestimating its complexity.
5:23 Redefining the Hardware-Software Contract: Traditional CPU design with its clear hardware-software contract for scalar operations is insufficient for AI, which is dominated by massive matrix multiplies and data movement in large arrays.
7:31 Deceptively Simple Operations: High-level AI operations in PyTorch appear simple, but beneath the surface, they involve huge, multi-dimensional tensor computations (e.g., 2000x2000x4000) that must be efficiently spread across numerous chips.
9:26 Transistors and Programming Paradigms: Keller highlights a trend: every ~100x increase in transistor count historically enables new programming paradigms, from scalar to vector to tensor processors, requiring new architectural approaches.
11:14 GPUs' Early AI Advantage: GPUs gained an early advantage in AI by effectively running "vectors of scalar programs," providing 5-10 times more mathematical power than traditional CPUs at the time.
13:50 The "Pyramid of Operation Rates": AI computation operates on a hierarchy: slow PyTorch instruction rates (hundreds of kHz) at the top, dispatching to faster chip-level operations, down to extremely high flop rates (e.g., 4,000 operations/second) at the hardware level. Managing this hierarchy is critical.
15:52 Compiler Strategy for AI: A key lesson learned is that building AI software effectively requires starting with fast, provable low-level kernels and then enabling their generation by compilers, rather than a top-down approach from high-level languages.
17:21 Cuda as a "Swamp": Keller controversially describes Nvidia's Cuda as a "swamp," implying it's a difficult, muddy territory to navigate for high performance, requiring extensive low-level optimization and "hacks."
17:59 AI Models: Small at the Top, Complex Beneath: Despite their power, many popular AI models (e.g., Llama, Stable Diffusion) are surprisingly concise at the PyTorch level (~600 lines of code) but rely on thousands of lines of highly optimized low-level kernel code.
18:43 Embracing Open Source: Tenstorrent has open-sourced its compiler (Buddha), bare-metal stack, and RISC-V architectural tests, believing in the value creation and talent attraction of open ecosystems.
19:24 Brain-Inspired Architecture: The human brain, operating at low frequencies (100-1000 Hz), achieves extraordinary computation through a highly organized "cortical column" structure, inspiring Tenstorrent's array-of-arrays approach for AI processors.
23:23 Scaling Challenges & Cost: While computation scales well with problem size, I/O bandwidth, power delivery, and cooling present non-linear scaling challenges, leading to exponentially rising costs for large, monolithic chips (e.g., IBM Power 5).
26:48 Pursuing Low-Cost AI Computing: Countering the trend of increasing AI hardware costs, Tenstorrent aims for linear cost with linear performance using a modular approach with smaller, interconnected AI processors and Ethernet for communication.
33:37 What Doesn't Work: Avoid "tricks" (e.g., exaggerated claims, single-bit computing), top-down auto-parallelizing compilers, and excessively large, expensive integrated packages.
36:49 What Really Works: Focus on solid semiconductor execution, strong math units, meticulous design, a clear split between runtime and compiler, sequential execution of large operations, data locality, Ethernet transport, and leveraging open source.
38:52 Future Directions: Tenstorrent plans to build a robust ML layer, improve debug tools (e.g., single-stepping millions of CPUs), innovate with chiplets for hardware configurability, enhance memory and compute, and release more open-source hardware and software specifications.
41:09 Conclusion: Keller celebrates the successful power-on of their new chip, emphasizing the blend of fundraising, open-source development, and hardware innovation as a "dumbest plan" that proved highly successful.