Abstract:

Jeff Hammond from NVIDIA provides an in-depth lecture on NVIDIA's key GPU communication libraries, NCCL and NVSHMEM. The talk addresses the fundamental differences between the two, explaining their distinct programming models and historical origins. Hammond contrasts the two-sided, message-passing paradigm of MPI (which influenced NCCL) with the one-sided, remote memory access (RMA) model of SHMEM (the basis for NVSHMEM), highlighting why MPI's general-purpose design presents challenges for high-performance GPU execution. Through a detailed matrix transpose benchmark on a DGX H100, he demonstrates the performance characteristics of various implementations. A key finding is the significant performance advantage (a ~30% speedup) of using NVSHMEM's pointer-based direct memory access over NVLink, which fuses communication and computation by eliminating explicit communication calls. The lecture connects these concepts to real-world AI applications, such as Mixture of Experts (MoE) models, where NVSHMEM's device-initiated, one-sided communication is critical for efficiently handling unstructured all-to-all patterns. Finally, Hammond discusses the future roadmap, revealing that upcoming versions of NCCL will incorporate one-sided and symmetric memory features, leading to a convergence of capabilities between the two libraries.

A Deep Dive into GPU Communication: NCCL and NVSHMEM

• 3:45 Historical Context: MPI vs. SHMEM: The talk begins by contrasting the two historical paradigms of high-performance computing communication.

  • MPI (Message Passing Interface): A two-sided model using send/receive pairs. It combines synchronization and data movement, is universally portable, but its generality can be a performance bottleneck on GPUs.
  • SHMEM (Symmetric Hierarchical Memory): A one-sided model using put/get operations. It decouples synchronization from data movement, enabling more efficient, low-overhead communication, especially for unstructured patterns.

• 5:12 NCCL for AI Workloads: NCCL (NVIDIA Collective Communications Library) is designed for the primary communication patterns in AI, such as all-reduce.

  • It supports all major parallelism types (data, tensor, pipeline, expert) and is integrated into all major AI frameworks (e.g., PyTorch).
  • Key takeaway: NCCL simplifies development by abstracting away complex collective algorithms, automatically selecting the best protocol based on hardware and message size.

• 8:12 NVSHMEM and Device-Initiated Communication: NVSHMEM implements a Partitioned Global Address Space (PGAS) model, allowing a GPU to directly access the memory of other GPUs.

  • Key takeaway: Its most unique feature is supporting device-initiated communication, allowing a CUDA kernel to send or receive data directly without returning to the host CPU. This drastically reduces latency for fine-grained communication.

• 38:18 Why NCCL is Not Just MPI on GPU: The speaker explains why MPI's full feature set is unsuitable for direct GPU execution. Features like complex data types, message tags, and wildcard receives (any_source) introduce logic and synchronization overhead that is inefficient for the GPU's parallel architecture. NCCL is a streamlined library that omits these features for performance.

• 43:02 Core Benchmark: Distributed Matrix Transpose: A matrix transpose operation is used as a benchmark to compare communication strategies. This operation naturally creates an all-to-all communication pattern, where every GPU must send data to every other GPU.

• 56:35 NCCL Performance Insights:

  • Using a group of send/receive calls to build a collective in NCCL performs as expected, saturating the NVLink switch bandwidth.
  • Attempting to manually manage communication with pairwise send/receive calls performs poorly. This is because NCCL's scheduler has its own flow-control logic optimized for collectives, and manual intervention interferes with it.
  • Key takeaway: The mantra is "trust NCCL." Let the library manage the collective implementation rather than trying to optimize it manually at a low level.

• 1:04:11 The Winning Strategy: NVSHMEM Pointer Access: The NVSHMEM implementation demonstrates a significant performance advantage.

  • By using nvshmem_ptr to get a direct pointer to remote GPU memory over NVLink, the benchmark can treat remote data as if it were local.
  • This eliminates explicit communication calls (put/get), fusing communication into the computation kernel.
  • Result: This approach is ~30% faster than the explicit communication methods, achieving 9.5 TB/s throughput on the benchmark.

• 1:23:25 Application in AI: Mixture of Experts (MoE): The concepts are directly applied to MoE models used by companies like Perplexity.

  • MoE requires an unstructured all-to-all pattern to route tokens to different "expert" models on various GPUs.
  • NVSHMEM is ideal for this, as its device-initiated, one-sided put operations allow the router kernel to send data directly without knowing the state of the receiving expert, dramatically reducing latency.

• 1:27:45 Future Roadmap: Library Convergence: The speaker reveals that the capabilities of NCCL and NVSHMEM are converging.

  • An upcoming version of NCCL will introduce symmetric memory allocation, remote pointer access over NVLink, and device-initiated APIs, bringing many of NVSHMEM's key advantages into the more widely used NCCL ecosystem.