This video provides a comprehensive introduction to multi-GPU programming, presented by Marcus Revak of Nvidia. The lecture explains the core motivations for using multiple GPUs—strong scaling (solving a fixed-size problem faster) and weak scaling (solving a larger problem in the same time)—using a 2D Jacobi solver as a running example. It details the concept of domain decomposition and the resulting need for "halo exchanges" to communicate boundary data between GPUs.
The presentation systematically introduces and compares three fundamental programming models:
MPI (Message Passing Interface): A standard for process-to-process communication. The talk covers its use for halo exchanges, the performance benefits of CUDA-aware MPI, and technologies like GPU Direct RDMA that allow network cards to access GPU memory directly, bypassing the CPU.
NCCL (NVIDIA Collective Communication Library): A library optimized for high-performance GPU communication. Its key advantage is being stream-aware, allowing communication operations to be enqueued on CUDA streams, which simplifies code and removes the need for explicit host-side synchronization.
NVSHMEM: A Partitioned Global Address Space (PGAS) library that enables one-sided communication (e.g., put operations without a matching receive) and, most significantly, allows communication to be initiated directly from within a CUDA kernel. This facilitates kernel fusion, reduces launch overhead, and can provide substantial performance gains in latency-sensitive applications.
The lecture concludes with a practical comparison of the libraries, outlining their primary use cases and emphasizing that they can be used together to build complex, high-performance applications.
An Introduction to Multi-GPU Programming Models
0:01:28 Why Use Multiple GPUs? The primary reasons are to solve problems faster (strong scaling) or to solve larger problems that don't fit in a single GPU's memory (weak scaling).
0:03:44 Example Application: The Jacobi Solver: The lecture uses a simple 2D Jacobi solver to demonstrate programming concepts. This iterative algorithm requires each point on a grid to be updated based on the values of its neighbors, creating a clear need for communication when the grid is split across multiple GPUs.
0:07:34 Domain Decomposition and Halo Exchange: To parallelize the problem, the data grid is split into chunks (domains), with each GPU handling one chunk. GPUs at the boundaries must exchange their edge data (a "halo") with their neighbors to perform correct calculations.
0:09:45 Introduction to MPI (Message Passing Interface): MPI is a standard library for communication between processes. A common model is to launch one MPI process per GPU. It provides primitives like MPI_Sendrecv to safely exchange halo data without causing deadlocks.
0:18:30 CUDA-Aware MPI and GPU Direct: Modern MPI implementations can work directly with GPU memory pointers. This enables technologies like GPU Direct RDMA, which allows a network card to transfer data directly from GPU memory to the network, significantly reducing latency and increasing bandwidth by bypassing the CPU.
0:34:15 Optimization: Overlapping Communication and Computation: A key technique to improve efficiency is to hide communication latency. By using separate CUDA streams, an application can start computing the boundary data, initiate the halo exchange, and then compute the large interior of its domain while the data transfer happens in the background.
0:38:47 Introduction to NCCL (NVIDIA Collective Communication Library): NCCL is a library optimized for GPU-to-GPU communication. Its main advantage is that its operations are stream-aware, meaning communication calls are enqueued on a CUDA stream just like kernels. This removes the need for explicit synchronization calls that are often required with MPI.
0:51:06 Introduction to NVSHMEM: A Partitioned Global Address Space (PGAS) library that enables two powerful concepts:
One-Sided Communication: A GPU can directly put data into another GPU's memory without the receiving GPU having to post a matching receive call.
In-Kernel Communication: Communication can be initiated directly from within a CUDA kernel. This allows for fine-grained, thread-level data transfers.
0:53:00 Benefit of NVSHMEM: Kernel Fusion: The ability to communicate from within a kernel allows developers to fuse multiple steps (e.g., computation, communication, more computation) into a single, persistent kernel. This eliminates kernel launch overhead and host-side synchronization, which is critical for strong-scaling applications that are sensitive to latency. An example from the QuDA application shows a 1.7x speedup from this technique.
MPI: The best starting point for porting existing CPU code or for general-purpose parallel programming.
NCCL: Ideal when you need highly optimized collectives (AllReduce, etc.) or want to simplify your code with stream-aware communication.
NVSHMEM: The choice for implementing advanced patterns like kernel fusion, overcoming latency bottlenecks in strong-scaling scenarios, or when a one-sided communication model simplifies the algorithm.
Key Takeaway: These libraries are not mutually exclusive and can be combined within the same application to leverage the strengths of each.