This video conducts an in-depth, head-to-head comparison of Nvidia's RTX 5090 and RTX Pro 6000 Blackwell GPUs, focusing on their performance in AI inferencing tasks. The analysis covers Large Language Model (LLM) inferencing using llama.cpp with the Gemma 3 model (Q4 and Q8 quantized) and image generation via ComfyUI with the Flux Context model (FP8).
The comparison highlights architectural similarities like the Blackwell microarchitecture, 5th-gen Tensor Cores, and FP8/FP4 support, while noting the RTX 6000 Pro's higher CUDA core count (24,064 vs. 21,760) and slightly superior FP32 throughput (125 TFLOPS vs. 105 TFLOPS). However, the most significant differentiator is VRAM: the 5090 features 32GB GDDR7, while the 6000 Pro boasts a massive 96GB GDDR7 with ECC support.
Performance benchmarks reveal that for LLM inferencing with Q4 quantized models that fit within 32GB, both GPUs offer very similar token generation and prompt evaluation speeds. The 96GB VRAM of the 6000 Pro becomes critical for loading larger Q8 quantized models (like Gemma 3 27B param), which directly led to an "out of memory" error on the single 32GB 5090. A multi-GPU setup with two 5090s on a PCIe 5.0 motherboard, however, demonstrated comparable Q8 performance to a single 6000 Pro. In image generation, the 6000 Pro consistently showed a noticeable performance edge, completing tasks roughly 22-25% faster than the 5090. The video concludes that while the 5090 offers a more budget-friendly, scalable multi-GPU solution, the 6000 Pro is indispensable for single-card workflows demanding immense VRAM capacity and professional-grade reliability.
Nvidia RTX 5090 vs. RTX Pro 6000 Blackwell: AI Inferencing Showdown
0:00:02 Introduction: The video presents a head-to-head comparison of Nvidia RTX 5090 and RTX Pro 6000 Blackwell GPUs, specifically for AI inferencing. The presenter mentions having multiple units of each card, totaling over 250GB of VRAM.
0:00:41 Video Scope: This video focuses on LLM inferencing with llama.cpp and text/image generation using ComfyUI. Video generation will be covered in a separate, upcoming video.
0:02:38 Architectural Foundation: Both GPUs are built on the Blackwell architecture, fabricated on a custom TSMC 4N process. The 5090 uses the GB202 monolithic die, while the 6000 Pro is based on a larger, more feature-complete Blackwell silicon, enabling higher core counts.
0:04:49 Shared & Differentiating Features: Both cards share 5th-gen Tensor Cores, 5th-gen Ray Tracing Cores, and support for FP8 and FP4 lower-precision math, crucial for AI performance and memory savings.
0:05:24 CUDA Cores/SMs: RTX 5090 has 21,760 CUDA cores (170 SMs), while the RTX 6000 Pro has 24,064 CUDA cores (188 SMs), representing a 10-11% increase for the Pro card.
0:06:20 FP32 Throughput: The 5090 achieves ~105 TFLOPS, and the 6000 Pro ~125 TFLOPS, giving the Pro a 10-15% edge in raw compute.
0:10:16 Bandwidth: Both utilize a 512-bit GDDR7 memory interface, achieving ~1792 GB/s bandwidth (70% higher than RTX 4090).
0:12:29 ECC Memory: The 6000 Pro features ECC memory for professional reliability, unlike the 5090's non-ECC memory.
0:12:45 Practical VRAM Impact: The 96GB VRAM of the 6000 Pro is essential for loading large AI models (e.g., 60GB+ Q8 quantized models like Gemma 3 72B), which would exceed the 32GB capacity of a single RTX 5090.
0:13:36 Test Rig Configuration: The system uses an Asus Pro WSW790E Sage SE motherboard with an Intel W790 chipset, supporting seven PCIe 5.0 x16 slots (enabling four double-wide GPUs at full x16 bandwidth). It's powered by an Intel Sapphire Rapids 56-core/112-thread CPU and 512GB ECC DDR5 RAM.
0:14:44 LLM Inferencing with llama.cpp (Gemma 3 Model):
0:18:01 Gemma 3 Q4 on RTX 5090: A large prompt (~9k tokens) was processed at 3,595 tokens/s (prompt evaluation) and 63.49 tokens/s (generation).
0:20:50 Gemma 3 Q4 on RTX 6000 Pro: Similar performance was observed, with 3,956 tokens/s (prompt evaluation) and 62.71 tokens/s (generation). For models fitting in VRAM, the performance difference was negligible.
0:25:09 Gemma 3 Q8 on RTX 6000 Pro: The Q8 model successfully loaded, requiring ~39GB VRAM, generating tokens at 39-41 tokens/s (slower due to higher precision, as expected).
0:27:19 Gemma 3 Q8 on single RTX 5090: Failed to load due to an "out of memory" error (32GB limit).
0:28:41 Gemma 3 Q8 on two RTX 5090s: The model was successfully loaded across two 5090s, utilizing 46GB VRAM. Performance was comparable to a single 6000 Pro (3,500 tokens/s prompt eval, 40 tokens/s generation), demonstrating the viability of multi-GPU setups with high-bandwidth PCIe 5.0.
0:31:12 Image Generation with ComfyUI (Flux Context FP8 Model):
0:32:51 On RTX 5090: Image generation tasks (e.g., "enhance details and make it higher resolution," "remove the woman") consistently took around 15.7-15.9 seconds after initial model loading.
0:36:08 On RTX 6000 Pro: The same tasks were completed noticeably faster, at approximately 12.4 seconds, indicating a roughly 22-25% performance improvement over the 5090 for this workload.
0:41:51 Conclusion: The primary distinction between the 5090 and 6000 Pro is VRAM capacity (32GB vs. 96GB). While performance for LLMs where models fit is nearly identical, the 6000 Pro's larger VRAM is crucial for single-card loading of high-precision, large models. For image generation, the 6000 Pro shows a clear performance advantage. Two 5090s offer a cost-effective alternative for VRAM-intensive tasks, but the 6000 Pro remains the choice for professional demands with single-card large model support and ECC memory.