Persona: Senior Silicon Architect and Hardware Systems Lead
Reviewers Recommendation
This material is essential for VLSI (Very Large Scale Integration) Engineers, Computer Architects, ML System Software Engineers, and Open-Source Hardware Researchers. It provides a high-fidelity case study on navigating physical area constraints and I/O bottlenecks when deploying custom AI accelerators on legacy process nodes.
Abstract
This lecture details the architecture, synthesis, and full-stack integration of "TinyTPU," a micro-scale Tensor Processing Unit (TPU) fabricated via the Tiny Tapeout project on the SkyWater 130nm process node. The design features a 2x2 systolic array optimized for 8-bit precision, occupying a footprint of approximately 320 x 100 micrometers. The presentation covers the transition from integer-based arithmetic to an FP8 (E4M3) execution model utilizing integer accumulation to conserve silicon area while maintaining numerical stability.
A primary technical focus is the mitigation of I/O serialization bottlenecks; the speaker illustrates a streaming data pattern that overlaps memory loading with computation to achieve a theoretical throughput of 100 MFLOPS at a 50 MHz clock frequency. The synthesis and verification workflow utilizes an open-source toolchain, including OpenRoad for physical layout and CocoTB for functional verification. Finally, the project demonstrates vertical integration by implementing a custom PyTorch 2.0 backend using TorchDynamo, enabling Quantization-Aware Training (QAT) and direct inference from a Python-centric environment.
Technical Summary: TinyTPU Architecture and Integration
0:00 – Introduction to Tiny Tapeout: The project leverages the Tiny Tapeout platform, a multi-project wafer (MPW) service that enables low-cost silicon fabrication on the SkyWater 130nm node. The hardware description is authored in Verilog and synthesized into GDSII using the open-source OpenRoad CAD flow.
3:17 – Physical Constraints and Scaling: The die area is restricted to approximately 320 x 100 micrometers (two tiles). Due to these constraints, the systolic array is scaled down to a 2x2 configuration. The speaker notes that while Google's TPUs utilize much larger arrays (e.g., 128x128), 2x2 is the fundamental building block from which larger matrix multiplications are decomposed.
8:06 – Systolic Array Processing Elements (PE): Unlike some architectures that shift partial sums, this design accumulates products locally within each PE to minimize area. Each PE performs an 8-bit multiply-accumulate (MAC) operation and passes weights (W) horizontally and inputs (X) vertically in a classic skewed data pattern.
10:53 – Overcoming I/O Bottlenecks: The chip is limited to 8-bit input and 8-bit output ports per cycle. To maximize throughput, the design implements a streaming pattern where computation begins before the full matrix is loaded (Cycle 6). By overlapping the output of 16-bit results with the loading of the next matrix, the system achieves a throughput of 2 FLOPS/cycle, or 100 MFLOPS at 50 MHz.
14:55 – FP8 Implementation Logic: Version 2 of the chip implements FP8 (E4M3) arithmetic. To reduce the complexity of floating-point addition, the design multiplies the FP8 mantissas into an 18-bit integer accumulator. This "fixed-point accumulation" strategy, similar to that used in NVIDIA H100 architectures, saves significant silicon area compared to full floating-point units. The final result is converted to BF16 for output.
22:22 – Verification and Static Timing Analysis (STA): Functional verification is conducted via CocoTB (Python-based testbench). The designer highlights the importance of STA to resolve setup/hold time violations and "antenna violations" (etching-induced capacitance issues) during the physical layout phase to ensure manufacturability.
38:32 – Software Stack & Matrix Kernels: The speaker details the development of C++ and Python kernels to handle matrix tiling, padding for odd-numbered dimensions, and stationary-weight optimizations. A "stationary weight" scheme was recently implemented to double throughput by keeping the weight matrix in the PE registers between operations.
44:43 – PyTorch 2.0 Integration: Integration is achieved through the PyTorch 2 Export (PT2E) workflow. The speaker utilizes Quantization-Aware Training (QAT) to simulate the loss of precision during training.
48:14 – Custom Torch.compile Backend: A custom backend was developed for torch.compile using TorchDynamo. This backend traverses the model computation graph, replaces standard linear operators with TinyTPU-specific matrix multiplication kernels, and fuses transpose/activation operations directly into the hardware calls.
51:57 – Key Takeaways: The project successfully demonstrates a "micron-scale" ML accelerator with a functional vertical stack. While performance is limited by the 130nm node and I/O bandwidth, the methodology confirms that open-source silicon tools can bridge the gap between high-level ML frameworks (PyTorch) and custom hardware.