This video explores co-packaged optics (CPO) as a critical innovation addressing the I/O bandwidth bottleneck in modern computing, particularly driven by the immense demands of generative AI. The speaker highlights how raw compute power (e.g., GPUs) has outpaced data transfer capabilities, creating an "I/O wall." The primary culprit is identified as the serializer/deserializer (SerDes) within traditional pluggable optical transceivers, which consumes increasing power and limits scalability. CPO offers a solution by integrating optical modules directly within the same package as switch silicon, minimizing electrical signal travel and improving efficiency. The video delves into the technical challenges of CPO, such as heat dissipation and light coupling, and traces its evolutionary path from onboard to near-package optics. It specifically examines TSMC's "Compact Universal Photonic Engine" (COUP) technology, which 3D-stacks electrical and photonic ICs. Crucially, the rise of AI is presented as the catalyst that transformed CPO from a niche concept to an urgent industry priority, leading to its adoption by major players like Nvidia and Broadcom. While CPO marks a significant advancement in bringing light and electrons closer, the video concludes that a fully photonic AI chip remains a distant prospect due to formidable technical and economic hurdles.
Summarizing "Co-Packaged Optics: The AI Revolution"
0:00 The I/O Bottleneck: The semiconductor industry has dramatically increased raw compute power (e.g., Nvidia B200 GPU's 178,000x flop increase over 1990s CPUs), but I/O bandwidth, particularly from optical transceivers, has not scaled as fast, creating a "memory wall" and "I/O wall" that prevents full utilization of compute power.
1:02 Server Rack Data Transfer: Data transfer in server racks involves silicon switch chips (directing data) and optics I/O (entering/exiting data). Electrical signals travel through copper wires for short distances (under 2 meters), but fiber optics are used for longer distances due to higher data capacity and fewer integrity issues.
1:51 Pluggable Optical Transceivers: The transition between electrical and light signals is managed by pluggable optical transceivers, which contain an "optical engine" (laser transmitter, modulator, photodiode receiver) and an "electrical engine" (handling signals from switch chips).
2:59 Moore's Law vs. Optics: While switch chips benefit significantly from Moore's Law (e.g., Broadcom's 102.4 Tbps switch chip), optical transceivers have grown much slower (e.g., 10 Gbps to 800 Gbps in 20 years), creating an inherent bottleneck in data centers.
4:07 The SerDes Power Problem: A major reason for optical I/O's underperformance is the Serializer/Deserializer (SerDes) component, which converts parallel electrical data to serial optical data. SerDes consumes a rapidly increasing proportion of power and area (e.g., 10-15% of switch power in 2013-14, now over twice that, and 10% of a GPU's total compute power). This power consumption limits available power for GPUs.
6:14 Introducing Co-Packaged Optics (CPO): CPO is the industry's response, aiming to integrate optical modules directly within the same package and on the same substrate as the switch silicon. This reduces the distance electrical signals travel over copper, improving efficiency and freeing up physical space on the faceplate.
7:39 Challenges of CPO: Significant challenges include heat dissipation (heat can warp materials and shift photonic waveguides' resonance wavelengths, making them stop working) and coupling issues (efficiently guiding light from larger optical fibers into tiny waveguides).
9:14 Industry Inertia & CPO Evolution: The industry's conservative nature and the established, cheap, and standardized pluggable transceivers created inertia. Intermediate steps included "onboard optics" (2018) – placing the optical engine directly on the PCB with switch chips – and "near-package optics" (2020) – using a high-speed silicon interposer between the chip and optical engine.
10:41 TSMC's COUP Technology: In 2021, TSMC introduced "Compact Universal Photonic Engine" (COUP), which 3D stacks an electrical IC on top of a photonic IC. This allows for advanced nodes on the electrical IC and cheaper nodes on the photonic IC, shortens inter-IC wiring, and claims significant power (40% less) and speed (170% faster) improvements.
12:36 Generative AI as the Catalyst: Initial adoption of COUP was slow, but the explosion of generative AI (post-ChatGPT) and the unprecedented power consumption of AI factories provided a strong economic and technical reason for vendors to finally adopt CPO. This spurred TSMC to accelerate COUP's roadmap.
13:45 CPO Adoption by Industry Leaders: Nvidia announced its first CPO products in 2025 with the Quantum X Infiniband and Spectrum X Ethernet platforms, claiming 3.5x better power efficiency, 10x better network resilience, and 1.3x faster deployment compared to pluggables. Broadcom is also a CPO pioneer (e.g., Tomahawk 5) and continues pushing switch chip speeds. Other foundries like GlobalFoundries (GF Photonic platform) and companies like IR Labs are also making significant advancements in silicon photonics.
15:57 The Future: Pure Photonic AI Chips? While CPO brings light and electrons closer, a complete takeover by light (pure photonic AI chips) remains a distant dream due to economic inertia, worsening heat and coupling issues in all-photonic chips, lack of scalable on-chip light sources, and difficulties in achieving certain logic gates optically. CPO is seen as a marvelous and necessary step driven by AI's impact on silicon supply chains.