This chapter, "How To Scale Your Model: How to Think About GPUs," delves into the architecture, networking, and scaling strategies for NVIDIA GPUs (e.g., H100, B200, GB200) in the context of large language models (LLMs), drawing comparisons with Google's TPUs. It dissects the internal workings of GPU chips, including Streaming Multiprocessors (SMs), Tensor Cores, CUDA Cores, and their memory hierarchies (HBM, L2, SMEM). A significant portion is dedicated to GPU networking, contrasting its hierarchical tree-based InfiniBand/NVLink fabric with TPU's toroidal mesh, and analyzing collective communication primitives (AllGather, AllReduce, AllToAll) within and across nodes. The text then provides a roofline analysis for common LLM parallelism techniques—Data, Tensor, Expert, and Pipeline Parallelism—identifying communication bottlenecks and practical batch size requirements. It concludes with a synthesis of optimal sharding recipes and acknowledges the practical discrepancies between theoretical and empirical GPU performance.
Scaling LLMs on NVIDIA GPUs: A Deep Dive
Core GPU Architecture (0:00):
Modern ML GPUs (H100, B200) consist of numerous Streaming Multiprocessors (SMs) specializing in matrix multiplication, connected to fast HBM (High Bandwidth Memory).
Each SM includes a dedicated Tensor Core (matrix multiplication unit), a Warp Scheduler (vector arithmetic unit with CUDA Cores), and a fast SMEM (L1 cache).
Unlike TPUs, which have a few powerful Tensor Cores, GPUs feature over 100 less powerful but independent SMs (e.g., 132 on H100).
CUDA Cores (SIMT model) offer flexible programming but can degrade performance with warp divergence, while Tensor Cores provide the vast majority of FLOPs/s (e.g., 990 bf16 TFLOP/s on H100).
Memory Hierarchy: Includes Registers, SMEM (256kB/SM on H100), a shared L2 Cache (~50MB on H100), and HBM (main memory, 80GB on H100, 192GB on B200). HBM bandwidth is 3.4TB/s on H100 and 8.0TB/s on B200.
GPU vs. TPU Chip-Level Comparison:
Modularity: GPUs are more modular with hundreds of small SMs/Tensor Cores, while TPUs have 1-2 large Tensor Cores.
Flexibility vs. Compiler Burden: GPUs offer more runtime flexibility, while TPUs place more burden on the compiler for optimal performance due to single-threaded control and vectorized instructions.
Cache Memory: TPUs generally possess more fast cache memory (VMEM) than GPUs (SMEM + TMEM), which is advantageous for LLM inference.
GPU Networking Overview (0:00):
GPUs employ a hierarchical tree-based switching network (NVLink within nodes, InfiniBand/Ethernet between nodes), contrasting with TPU's 2D/3D torus topology.
Intra-Node Networking and Collectives (0:00):
Node Structure: Typically 8 GPUs (up to 72 for GB200) are fully connected within a node via NVLinks and NVSwitches, offering low latency and high bandwidth.
H100 Node Bandwidth: Each GPU in an 8xH100 node has 450GB/s of full-duplex bandwidth into the NVLink network.
AllGather/ReduceScatter: Cost is approximately Bytes / (GPU egress bandwidth). For 8xH100, this is B / 450e9.
AllToAll: More efficient within a node due to all-to-all connectivity, costing roughly B / (8 * GPU egress bandwidth) for 8 GPUs.
Empirical Performance: Actual H100 AllReduce bandwidth often falls around 370GB/s, not the theoretical 450GB/s.
SHARP (Scalable Hierarchical Aggregation and Reduction Protocol): NVIDIA switches can perform "in-network reductions" to theoretically halve AllReduce costs, but practical gains are closer to 30%.
Cross-Node Networking and Collectives (0:00):
Scalable Units (SUs) & SuperPods: Nodes are grouped into SUs (e.g., 32 nodes/256 GPUs) connected via InfiniBand, which in turn connect to spine switches for larger SuperPods (e.g., 1024 H100 GPUs).
Fat Tree Topology: The InfiniBand network is designed as a fat tree, guaranteeing full bisection bandwidth beyond the node level (e.g., 400GB/s node-to-node on an H100 DGX SuperPod).
Collective Cost: AllGather/ReduceScatter/AllReduce costs roughly Bytes / (Node egress bandwidth). For H100, this is B / 400e9.
AllToAll Degradation: Cross-node AllToAlls are significantly more costly, as they cannot leverage the full bisection bandwidth effectively for every pair, leading to much lower effective bandwidth (e.g., 50GB/s vs. 400GB/s).
GB200 NVL72 Impact: These new clusters drastically increase node size (72 GPUs) and node egress bandwidth (3.6TB/s), potentially shifting communication bottlenecks.
Rooflines for LLM Scaling on GPUs (0:00):
Data Parallelism (DP) / ZeRO Sharding: Requires a high per-GPU token batch size (around 2500 tokens for H100/B200) to be compute-bound, which is considerably higher than TPUs (~850 tokens). MoE models inflate this requirement by a factor of E/k.
Tensor Parallelism (TP): Becomes communication-bound when the sharding axis Y is greater than F / 2475 (approx), generally limiting TP to intra-node or at most 2-node configurations due to bandwidth constraints.
Expert Parallelism (EP): Feasible if F (feed-forward dimension) is large or if the degree of expert parallelism is small, otherwise limited by communication.
Pipeline Parallelism (PP): Has very low communication cost per hop, making it theoretically efficient. However, it introduces significant code complexity, "pipeline bubbles," and can negatively impact data parallelism (especially ZeRO-3) by requiring frequent AllGathers.
Practical Scaling Examples:
DeepSeek V3: Trained on 2048 H800 GPUs using 64-way EP, 16-way PP, and 2-way ZeRO-1 DP, achieving a large batch size due to model sparsity.
LLaMA-3: Trained on 16,000 H100s with 8-way TP, 16-way PP, and 128-way ZeRO-1 DP. PP helped reduce the effective cost of data parallelism.
Key Takeaways for LLM Scaling on GPUs:
Data parallelism demands high local batch sizes.
Tensor parallelism is largely confined to single NVLink domains.
Mixing PP, EP, and TP is often necessary to span many nodes and amortize communication costs.
Pipeline parallelism is communication-efficient but complex to implement optimally and often incompatible with ZeRO-3.