This video demonstrates how to run the massive 685-billion parameter DeepSeek V3 0324 large language model on a single high-end workstation. The tutorial outlines a method using an Nvidia RTX 4090 GPU, a powerful multi-core CPU (Intel Xeon 8480+ with 512GB RAM), and the K Transformers framework within a Docker environment. The key enabler is a hybrid quantization approach: FP8 (8-bit floating point) precision is used for the model's critical attention and shared layers on the GPU, while 4-bit GGML quantization is applied to the less frequently accessed Mixture of Experts (MoE) layers, offloaded to the CPU. This innovative strategy allows the model to leverage GPU acceleration where it matters most, minimize VRAM usage, and exploit high CPU core counts for efficient inference, making an ultra-large, open-source model accessible on consumer-grade (albeit high-end) hardware with a 32K context limit and an impressive 10 tokens/second throughput.
Running DeepSeek V3 0324 (685B Parameters) on a Single Workstation
0:03 Goal: The video demonstrates how to run the DeepSeek V3 0324, a 685 billion parameter Mixture of Experts (MoE) Large Language Model (LLM), on a single workstation with an Nvidia RTX 4090 GPU and a powerful CPU with ample RAM.
0:38 DeepSeek V3 0324 Capabilities: This latest DeepSeek version significantly improves reasoning and coding benchmarks, supports function calling, structured JSON outputs, and tool invocation, rivaling top closed-source models while remaining open-source. Its FP8 tensor technology reduces memory usage, making it accessible on standard hardware.
1:27 Hardware Setup: The setup used includes an RTX 4090, an Intel Xeon 8480+ processor (56 cores, 112 threads), 512GB of DDR5 ECC RAM, highlighting that this is a single workstation, not a multi-GPU cluster.
2:38 Mixture of Experts (MoE) Architecture: DeepSeek V3 is an MoE model with nearly 700 billion parameters, but only a subset (~37 billion) is active for any given token. This design provides huge capacity without needing to load all weights simultaneously, which is crucial for limited hardware.
3:51 FP8 + GGML Hybrid Quantization: This is the core technique to fit the model on a 24GB GPU.
FP8 Precision (GPU): Attention mechanisms and shared layers (passed by every token) run in 8-bit floating-point on the GPU using custom K Transformers kernels, drastically reducing memory usage while maintaining accuracy and speed. Requires FP8-compatible GPUs (RTX 4090/4080).
4-bit GGML Quantization (CPU): The expert layers, comprising the bulk of the 685 billion parameters, are kept in 4-bit quantized GGUF format and stored in CPU memory. Only necessary expert weights are loaded and computed on the CPU with high parallelism, requiring hundreds of gigabytes of system RAM (512GB in this setup).
7:25 K Transformers Framework: This open-source inference framework from KVK AI orchestrates the complexity. It natively supports FP8 GPU kernels, automatically manages data movement between CPU and GPU, and uses multi-threading to leverage all available CPU cores for expert layers.
8:31 32K Context Limit: The K Transformers Docker image provides an extended 32K context limit, allowing for very long documents or complex chat histories.
9:18 Docker for Simplified Setup: An official K Transformers Docker image (optimized for AVX-512 CPUs like the Xeon 8480+) packages all necessary dependencies (CUDA, PyTorch, K Transformers), eliminating compilation and environment issues, ensuring a reproducible setup.
10:52 Step-by-Step Implementation: The tutorial details creating project directories, downloading FP8 weights from DeepSeek AI and Q4KM GGUF from Unsllo, pulling the correct K Transformers Docker image, and merging the models using a Python script within Docker.
13:54 Running the Server: Instructions are provided for running the k_transformers server command inside the Docker container, specifying model paths, the FP8 optimization configuration (YAML), and CPU inference threads (e.g., 57 for a 56-core CPU). The first run involves JIT compilation, leading to a brief warm-up time.
20:00 Performance and Outcomes: The setup achieved approximately 10 tokens per second throughput with under 24GB of VRAM usage, an impressive feat for such a massive model. The system also supports an extended 32K context.
21:46 Opportunities & Accessibility: This method makes ultra-powerful, open-source LLMs like DeepSeek V3 0324 accessible for hosting chatbots, local AI research, and integration into applications without relying on external APIs, particularly for users with high-end single workstations.