This video features a conversation with Nir Shavit, a professor at Technion and MIT and the founder of Neural Magic, about the concept of sparsity in neural networks. Shavit argues that current AI development, which relies on dense computations on power-hungry GPUs, is fundamentally opposite to the highly efficient and sparse nature of the human brain.
His company, Neural Magic, leverages sparsity to enable standard CPUs to achieve GPU-level performance for model inference. This is accomplished by first pruning models to drastically reduce the required computations (flops), and then employing a proprietary execution strategy called "tensor columns." This method runs networks "depth-wise" to maximize the use of the CPU's fast cache hierarchy, overcoming the bottleneck of slower main memory. Shavit explains that this approach allows for running larger, more accurate models (e.g., a pruned BERT-large) faster than smaller, dense models, making high-performance AI more accessible and efficient, particularly for edge devices. The discussion also touches on the future of AI architecture, envisioning a shift away from rigid layers toward more dynamic, asynchronous, and graph-like computations.
Making CPUs Competitive: A Conversation on Sparsity in Neural Networks
00:03:26 Biological Inspiration for Sparsity: The human brain is described as a "cell phone of compute on a petabyte of memory," meaning it is extremely sparse and energy-efficient. In contrast, current GPU-based AI is the opposite, using immense computational power with limited memory. Shavit argues that sparsity is a key principle to achieve brain-like efficiency in AI.
00:04:22 The Promise of Neural Magic: The company was founded on the discovery that CPUs can match GPU speeds for AI inference. By using sparsity and other software optimizations, they aim to eliminate the need for specialized, expensive hardware accelerators.
00:09:34 Don't Replicate, Learn: Shavit advocates for implementing the brain's algorithms, such as sparsity, rather than trying to replicate its exact physical architecture in silicon. Silicon-based processors have different strengths that should be leveraged.
00:21:58 The Power of Pruning Larger Models: The key advantage of sparsity isn't just making a model smaller. The best strategy is to take a larger, more accurate model (e.g., BERT-large), prune it heavily, and achieve performance that is both faster and more accurate than a smaller dense model (e.g., BERT-base).
00:25:41 CPUs are Better for Sparsity: GPUs, with their lockstep parallel design, are ill-suited for the irregular computations of sparse models. CPUs, with their powerful cores, sophisticated cache hierarchies, and access to large memory pools, are a more natural fit.
00:30:18 Neural Magic's "Tensor Columns": To achieve high performance, Neural Magic uses a technique to run networks "depth-wise." This keeps the active computation within the CPU's fast L1/L2 cache, avoiding the main memory bottleneck. This allows a 4-core CPU to match the latency of an NVIDIA A100 GPU on a sparse BERT model.
00:34:04 The Future of AI Architectures: The future of AI may move beyond rigid, layer-by-layer execution. Shavit envisions more dynamic models where computations flow through asynchronous paths, a paradigm that is well-suited for CPUs.
00:45:37 From Data Centers to the Edge: Neural Magic's software is designed for a wide range of applications, from large data centers to edge devices. They are actively expanding support for ARM architectures to bring high-performance AI to smartphones, security cameras, and robots.
00:47:09 Easy to Get Started: The company's entire software stack is open source. Developers can get started by using pre-made "recipes" to apply sparse transfer learning to existing models, making the transition to efficient, CPU-based inference straightforward.