This video presents a comparative analysis of high-end consumer GPUs for running local large language models (LLMs). The host benchmarks the new Nvidia RTX 5090 against AMD's flagship RX 7900 XTX, and later includes the Nvidia RTX 5080, to determine the best value for performance. The tests are conducted on identical systems running both Windows and Linux, using popular LLM runners like LM Studio and Olama with various models. Key performance metrics such as token generation speed and prompt processing speed are analyzed. The video highlights not only raw performance differences but also crucial factors like VRAM capacity, memory bandwidth, power consumption, and operating system impact, culminating in a critical "gotcha" moment that underscores the importance of VRAM size for handling larger models.
GPU Showdown: Nvidia vs. AMD for Local AI
0:00 - Introduction: The Quest for Speed: The video aims to determine the best GPU for running local LLMs by comparing the brand-new, expensive Nvidia RTX 5090 (32GB VRAM, >$3000) with the previous-generation flagship AMD RX 7900 XTX (24GB VRAM, <$1000).
1:50 - Key Hardware Factors: Performance for local LLMs is heavily influenced by VRAM capacity (the model must fit entirely) and memory bandwidth. The 5090 has a significant bandwidth advantage (1700 GB/s vs. 960 GB/s for the 7900 XTX).
4:00 - Initial Windows Benchmarks (LM Studio): Using a small model (Gemma 3), the RTX 5090 is significantly faster than the AMD card (~196 vs. 112 tokens/sec). Surprisingly, using the Vulcan backend on the Nvidia GPU was slightly faster than using its native CUDA backend.
7:22 - Olama Benchmarks (Windows): Testing with Olama reveals a strange anomaly: while the RTX 5090 is much faster at generating tokens, the AMD RX 7900 XTX consistently shows a much faster prompt processing speed. The Nvidia card also produced some CUDA errors.
9:21 - Linux Performance: Switching to Linux, the performance gap widens. The RTX 5090 is 2.5 times faster than the AMD card for token generation (~248 vs. 99 tokens/sec) and also takes the lead in prompt processing, resolving the anomaly seen on Windows.
11:08 - Large Model Test (Quen 2.5 Coder): A larger 20GB model pushes the hardware harder. The RTX 5090 draws a massive 775 watts under load. It maintains a strong lead in generation speed (62 vs. 25 tokens/sec), but the AMD card again shows a surprisingly high prompt processing rate.
13:32 - The RTX 5080 Comparison: To test the impact of memory bandwidth, the RTX 5080 (which has the same 960 GB/s bandwidth as the AMD card) is introduced. It outperforms the AMD on small models.
14:50 - The VRAM "Gotcha": When running the large 20GB model, the RTX 5080's performance plummets to just 8 tokens/sec. Key Takeaway: The 5080 only has 16GB of VRAM, so the model "spills over" to the much slower system RAM and CPU. This demonstrates that having sufficient VRAM is more critical than other specs for running large models.
16:00 - VRAM is King: The cheaper AMD RX 7900 XTX, with its 24GB of VRAM, easily outperforms the more expensive RTX 5080 on the larger model, highlighting its superior value for users wanting to run bigger, more capable LLMs.
16:56 - Practical Application: The host briefly shows the local LLM setup integrated into VS Code, demonstrating a real-world use case for developers.