Abstract:

This presentation outlines the evolution of Google’s TPU SparseCore (SC) from a niche embedding accelerator into a general-purpose offloading unit for irregular computations. While Tensor Cores handle dense matrix multiplications, the SparseCore architecture—a tiled vector processor—is optimized for hypersparse data, irregular memory access (gathers/scatters), and data summarization (sorting, histograms).

The XLA compiler team demonstrates that offloading these "overhead" ops to SparseCore achieves speedups ranging from 25% in LLMs to 80% in perception models by pipelining executions and hiding latency. However, full Open Source Software (OSS) availability is delayed until 2026, with current access limited to internal Google workloads and an "Alpha Partner" program. As far as I can tell, the primary bottleneck for external users isn't the hardware, but the stability of the Mosaic/Palace programming models and XLA’s automated partitioning logic.

SparseCore Offloading: Maximizing TPU Utilization via Heterogeneous Execution

• 0:33 Functional Division: Tensor Cores (TC) are "workhorses" for dense math; SparseCores (SC) are specialized for everything else to prevent hardware underutilization. If you aren't using SC, you’re wasting silicon.
• 1:59 SparseCore Strengths: Specifically designed for 99.99% sparsity, irregular memory patterns (gather/scatter), and hardware-native summarization (sort, prefix sum).
• 3:07 Tiled Architecture: SC uses a tiled vector core (8 or 16-lane SIMD) with a centralized sequencer and software-managed scratchpad memory. This reduces latency compared to hitting HBM directly from the TC.
• 4:33 Hardware Generations: Newer "Ironwood" chips double the SIMD width compared to "Trillium." I estimate the probability of these being available for non-Google Cloud customers before mid-2026 is less than 20%.
• 6:33 Performance Benchmarks: In microbenchmarks, SC is magnitudes faster than TC for element-wise and lane-wise gathers. For data formatting, SC is 50–75% faster.
• 10:04 XLA Compiler Role: XLA performs the heavy lifting by translating high-level graphs into parallel executables for both cores. It uses "one-step delay gradients" and pipelining to hide SC latency behind TC compute.
• 13:09 Internal Success Stories: Internal Google recommendation models saw a 50% step time reduction; LLMs saw a 25% speedup by offloading communication/collectives to SC.
• 14:42 Implementation Methods: Users can currently trigger SC offloading via XLA flags, explicit annotations (TPU sparse_core), or experimental custom kernels via "Palace."
• 18:01 Alpha Partner Program: Google is seeking external partners for 2025 to test these features. They claim a target of 10% performance improvement, which seems like a conservative "floor" given the internal 25-80% claims.

Reviewer Group: ML Infrastructure Architects and Compiler Engineers (e.g., NVIDIA/AMD performance teams or PyTorch Core devs).

Their Perspective Summary: "Google is finally opening up the TPU’s co-processor to stop the 'Tensor Core bottleneck' in non-standard layers. The hardware is essentially a specialized SIMD array with better address generation for sparse lookups. The 80% throughput gains in perception models are impressive but likely reflect how poorly the Tensor Core handles gathers rather than how 'magical' the SparseCore is. We’re skeptical of the 2026 OSS timeline—Google historically keeps the best XLA partitioning heuristics internal until the hardware is N-1. If you're on TPU, you need to be in the Alpha program now or you're overpaying for idle silicon."