ANALYZE AND ADOPT: SENIOR MACHINE LEARNING INFRASTRUCTURE ENGINEER
As a Senior ML Infrastructure Engineer specializing in local LLM (Large Language Model) deployment and hardware optimization, I have synthesized the performance data provided in the transcript. This analysis focuses on the throughput metrics (tokens per second), quantization impacts, and hardware architecture bottlenecks relevant to deploying agentic workflows on consumer-grade silicon.
SUMMARIZE (STRICT OBJECTIVITY)
Abstract:
This technical benchmark evaluates the performance of the Qwen 3.6 35B Mixture of Experts (MoE) model against Gemma 4 and earlier Qwen iterations across a diverse range of NVIDIA hardware (RTX 3060, 5060 Ti, 3090, 4090). The analysis highlights a significant performance paradigm shift facilitated by "sparse" MoE architectures, which offer substantially higher throughput for tool-calling and agentic use cases compared to traditional dense models.
Key technical findings demonstrate that Qwen 3.6 (A3B MoE) achieves up to 8,000+ tokens per second (TPS) in prompt processing on flagship hardware. The report emphasizes the critical necessity of "VRAM-fitting"—ensuring the model footprint resides entirely within GPU memory—to avoid catastrophic performance degradation caused by PCIe bus latency, particularly when utilizing 1x risers or system RAM spillover. While Gemma 4 Sparse recorded the highest raw throughput (10,000 TPS), Qwen 3.6 is identified as a superior candidate for local agents due to its purported advancements in tool-calling accuracy.
Benchmarking Qwen 3.6 and Gemma 4: Local AI Infrastructure and Throughput Analysis
0:00 Agentic Optimization: Qwen 3.6 is positioned as a primary expert for local agentic workflows, addressing the high-latency criticisms of previous dense models like Qwen 3.5 27B and Gemma 31B.
1:13 Entry-Level Scaling (Dual 3060s): Utilizing dual RTX 3060 12GB cards provides a 24GB VRAM pool, sufficient for Q4 quantization. Prompt processing on this setup peaks at ~3,200 TPS for sparse models compared to ~750 TPS for dense architectures.
3:05 Batching and Prompt Processing: High-intensity agentic workflows require robust prompt processing. Benchmarks show a significant acceleration in throughput as context size increases from 128 to 8K, peaking before slight declines at 16K.
5:09 Sparse vs. Dense Performance Gap: Dense models (Qwen 3.5, Gemma 31B) exhibit substantial throughput degradation, often yielding only 16–20% of the speed of sparse MoE counterparts on identical hardware.
8:11 Hardware Architecture and PCIe Bottlenecks: Testing on 5060 Ti (16GB) cards reveals that while MoE models are fast, exceeding VRAM capacity triggers a massive performance collapse (dropping from 3,500 TPS to ~118 TPS) due to 1x PCIe riser bandwidth limitations (~1GB/s).
14:39 Multi-GPU Scaling and Diminishing Returns: Scaling from one to four RTX 3090s demonstrates that while dual-GPU setups increase prompt processing (from ~3,900 to ~5,300 TPS), quad-GPU configurations introduce latency that can actually reduce performance for smaller 35B models.
17:34 VRAM Residency Criticality: Even on high-bandwidth Gen4 x16 interfaces, sharding models over to system memory results in significant performance hits, preventing the GPU from reaching full utilization (capping at ~60%).
20:00 Flagship Performance (RTX 4090): The 4090 achieves "smoky fast" results, with Qwen 3.6 peaking at ~8,189 TPS for prompt processing and 175 TPS for text generation at Q4 quantization.
24:00 Throughput Peak (Gemma 4 Sparse): Gemma 4 Sparse achieves the record throughput of 10,000 TPS on the RTX 4090, though the speaker notes potential trade-offs in tool-calling reliability compared to Qwen.
28:00 Optimal Deployment Strategy: For local agents (e.g., Hermes), the recommended configuration is a Q4 quantization of Qwen 3.6 35B on a single 4090 or dual 3090/3060 setup, targeting a 64K–128K context window fully contained in VRAM.