This video is a live-streamed technical demonstration and benchmark of the K Transformers inference engine, focusing on its ability to run the 671-billion-parameter DeepSeek R1 model on a home AI server with heavy CPU offloading. The host, Ahmed, walks through the entire process, from building the engine from source and troubleshooting CUDA compilation errors to successfully loading the 4-bit quantized model. The model is run using a single RTX 3090 GPU for its active parameters, while the vast majority is offloaded to the system's 512GB of RAM. A direct performance comparison is made against the popular Llama.cpp engine under identical conditions. The results validate K Transformers' claims, showing a dramatic improvement in both prompt evaluation speed (time-to-first-token) and generation rate (tokens per second), confirming its effectiveness for running exceptionally large models on consumer-grade hardware through optimized CPU/RAM utilization.
Running the 671B DeepSeek Model: A K Transformers Live Test
0:01:00 Introduction & Goal: Host Ahmed, accompanied by Osama, begins his first live stream with the goal of testing the newly optimized K Transformers inference engine to run the massive DeepSeek R1 671B parameter model.
0:04:04 Hardware Showcase: Ahmed presents his home AI server, the platform for the experiment, which is equipped with 14x RTX 3090 GPUs (336 GB total VRAM) and 512 GB of DDR4 system RAM.
0:07:26 Explaining Inference Engines & CPU Offloading: The hosts provide a high-level explanation of what inference engines (like Llama.cpp) do and define the key concept of CPU offloading, which involves loading parts of a large model into the computer's main RAM when it's too big to fit into the GPU's VRAM.
0:13:36 Understanding Quantization: Ahmed and Osama discuss model quantization, a compression technique that reduces the precision of a model's weights (e.g., from 32-bit to 4-bit) to significantly shrink its size, making it possible to run on less powerful hardware.
0:39:35 The Build Process & Troubleshooting: The practical demonstration begins as they attempt to build K Transformers from source. They encounter and successfully troubleshoot compilation errors related to missing CUDA library dependencies.
1:39:23 First Successful Run with K Transformers: After a successful build, they launch the 4-bit quantized DeepSeek R1 model. It loads successfully by offloading approximately 380 GB to the system RAM and using just 13 GB of VRAM on a single GPU.
1:43:39 Live Test & Impressive Performance: They test the running model with a prompt ("How many Rs are in the word strawberry?"). The model responds quickly and correctly, generating text at a very usable speed of around 8 tokens per second, proving the viability of the setup.
2:52:54 Building Llama.cpp for a Direct Comparison: To validate the performance claims, Ahmed begins building the Llama.cpp engine from source to run the exact same model under the same conditions.
3:11:41 Full GPU Inference Demonstration: While Llama.cpp compiles, Ahmed provides a performance contrast by running other models fully on multiple GPUs using different engines (vLLM, ExLlamaV2), showcasing much higher speeds of 30-65+ tokens per second.
4:32:08 The Final Benchmark: K Transformers vs. Llama.cpp: With Llama.cpp running the DeepSeek R1 model, the direct comparison is made. Llama.cpp achieves a generation speed of ~4.45 tokens/second. In contrast, K Transformers consistently performs at ~7.5-8.5 tokens/second, nearly doubling the speed. Prompt evaluation speed is even more dramatic, jumping from Llama.cpp's 0.8 tokens/sec to K Transformers' 11-13 tokens/sec.
4:51:20 Conclusion & Key Takeaway: The experiment is a success. The stream concludes that K Transformers delivers on its promise, providing a significantly faster and more efficient solution than Llama.cpp for running extremely large language models that require heavy CPU and RAM offloading.