Abstract:

This video details the unboxing, setup, and benchmarking of a dual NVIDIA RTX 6000 Ada Lovelace (Blackwell) workstation GPU setup within a high-end AI rig. The presenter unboxes the second GPU, highlighting its anti-tamper seal, and integrates both 96GB VRAM cards into a system featuring an ASUS WSW Pro 790E Sage motherboard, 512GB DDR5 RAM, and a 56-core Intel Xeon Sapphire Rapids engineering sample CPU running Ubuntu 24.04.

Performance is evaluated across three large language models—Quan 3 235B Q4KM, Deepseek R1 67B, and Llama 4 Maverick IQ3—using llama.cpp and ik_llama.cpp frameworks. The Quan 3 model, fully offloaded to the GPUs, demonstrates exceptional speed (58 tokens/sec generation, 1318 tokens/sec prompt processing) even with a 128K context length. In contrast, the Deepseek R1 model, requiring partial GPU offloading due to its size, showed significantly lower performance (12-15 tokens/sec generation) despite higher CPU utilization. Llama 4 Maverick, also partially offloaded, provided good generation speed (47-57 tokens/sec) but inconsistent code quality. The video concludes with a discussion on the trade-offs between cost, performance, VRAM capacity, and privacy for local AI inferencing, suggesting that while the Blackwell GPUs offer immense power, a sweet spot for performance-per-cost might lie with consumer-grade cards like the RTX 5090 or 4090 for many use cases.

Benchmarking Dual NVIDIA RTX 6000 Blackwell GPUs for AI Inferencing

• 0:00 Unboxing Second RTX 6000 Blackwell GPU: The video begins with unboxing the second NVIDIA RTX 6000 Ada Lovelace (Blackwell) workstation GPU, noting its 96GB VRAM.
• 0:23 Anti-Tamper Seal Demonstration: The presenter showcases a clever anti-tamper seal on the GPU packaging, where the word "VOID" becomes visible upon opening, indicating if the product has been tampered with.
• 4:34 Initial Setup and System Overview: The two Blackwell GPUs are installed in an AI rig running Ubuntu 24.04. The rig's specifications include an ASUS WSW Pro 790E Sage motherboard, 512GB DDR5 RAM, and a 56-core/112-thread Intel Xeon 4th Gen Sapphire Rapids engineering sample CPU (QYFS).
• 5:26 GPU Detection and VRAM Confirmation: nvidia-smi command confirms successful detection of both GPUs, each with 96GB of VRAM, totaling an "amazing amount" of video RAM.
• 5:58 Quan 3 235B Q4KM Model Test (llama.cpp):
  • The entire Quan 3 235B Q4KM model (35 billion parameters) is successfully offloaded to both GPUs.
  • 11:11 Performance: Achieves very fast generation speeds of 58 tokens per second and an impressive 1318 tokens per second for prompt processing.
  • 11:24 Context Length: Tested with 70K and later 128K context length (using a specialized model version).
  • 14:39 Power Consumption: The entire rig draws approximately 967 watts during peak operation.
  • Key Takeaway: Fully offloading smaller, high-context models like Quan 3 results in exceptional speed.
• 21:12 Deepseek R1 67B Model Test (llama.cpp):
  • This larger model (67 billion parameters) is too big for full GPU offloading with 96GB VRAM per card.
  • 26:47 Layer Offloading Challenges: Initial attempts to offload 40 layers failed due to memory limitations; extensive experimentation was required to find a stable configuration (eventually offloading 22 layers to each GPU).
  • 33:56 Performance: Achieves a significantly slower 12-15 tokens per second generation speed and approximately 100-150 tokens per second for prompt processing.
  • 32:09 Power Consumption: Draws around 835 watts from the wall during mixed CPU/GPU load.
  • Key Takeaway: Partial offloading of larger models leads to lower performance and higher CPU utilization, making it less impressive than full GPU offloading. Deepseek generated a more feature-rich but slower snake game.
• 44:39 Llama 4 Maverick IQ3 Model Test (ik_llama.cpp):
  • The Llama 4 Maverick IQ3 model (32 billion parameters) is partially offloaded to the GPUs (optimized to 29 layers per GPU for a total of 58 layers).
  • 59:38 Performance: Shows good generation speed of 15-57 tokens per second and prompt processing speeds ranging from 152-351 tokens per second.
  • 1:21:10 Code Quality: Generated a functional but initially basic snake game, requiring prompts for improved features and aesthetics.
  • Key Takeaway: ik_llama.cpp offers good performance for partially offloaded models, but the Llama 4 Maverick's code quality was less consistent than Quan 3 or Deepseek.
• 1:00:49 General Discussion & Conclusion:
  • GPU vs. CPU Load: Fully offloaded models (like Quan 3) are GPU-bound, maximizing GPU power. Partially offloaded models involve significant CPU work, leading to higher overall system power draw and lower GPU utilization percentages.
  • Power & Cooling: The rig consumes significant power (~800-970W) and requires industrial-grade cooling (Noctua fans in the basement due to noise). Future plans include moving the rig to a garage with a 2000W dedicated power line.
  • Cost vs. Performance: The current setup costs upwards of $20,000-$25,000. The presenter questions if such high-end GPUs are always "worth it" compared to more affordable alternatives like two RTX 5090s or four RTX 4090s (around $10,000 for 96GB VRAM), which might offer a better performance-per-cost for most use cases.
  • Privacy: The primary motivation for building such a powerful local AI rig is to ensure local and private processing of sensitive information, rather than relying on cloud APIs.