A group of Machine Learning Engineers, AI Research Scientists, and LLM Optimization Experts would be the ideal audience to review this material. They would focus on the architectural distinctions between Mixture of Experts (MoE) and dense models, the practicalities of local quantization (Q8/Q4), and the real-world inference performance (tokens per second) on enterprise-grade hardware like the DGX Spark.
Abstract:
This analysis evaluates the high-end release of the Gemma 4 family, specifically comparing the 26B Mixture of Experts (MoE) model (4B active parameters) and the 31B Dense model. Testing was conducted on a DGX Spark local system using Q8 quantization for the MoE variant, while the 31B Dense model was assessed via Nvidia NIM APIs due to current local quantization instabilities. The evaluation covers cross-domain capabilities including JavaScript-based game engine generation, multimodal "wireframe-to-code" translation, and zero-shot visual reasoning. Results indicate that while both models exhibit state-of-the-art reasoning for their size, the 26B MoE variant demonstrates superior efficiency and aesthetic output in multimodal tasks, achieving high inference speeds (22+ TPS at Q8) suitable for local deployment.
00:00 Gemma 4 Model Family Overview: Google has released four sizes of Gemma 4; this evaluation focuses on the 31B Dense (256K context) and the 26B MoE (4B active, 256K context) models. Both are released under the Apache 2.0 license.
02:37 Benchmark Claims: Preliminary data suggests these models perform comparably to significantly larger models (e.g., GLM5 or K2.5) while utilizing roughly 30x fewer parameters.
04:25 Local Deployment & Quantization: The 26B MoE model runs effectively on local hardware (DGX Spark) at Q8 quantization. However, the 31B Dense model currently faces configuration or quantization bugs (weird characters/language switching), necessitating testing via Nvidia NIM cloud APIs.
26B MoE: Initially produced a minimalist UI, but showed high "instruction following" capability by dramatically improving aesthetics, adding translucency, and a theme engine upon receiving critical feedback.
31B Dense: Generated "Nova OS," featuring a functional clock and an integrated "clicker quest" game logic.
13:00 3D Scene and FPS Generation: Both models successfully utilized Three.js for procedural 3D generation.
26B MoE: Developed "Subway Protocol," including weapon animations and muzzle flashes.
31B Dense: Produced "Subway Survival," featuring more advanced lighting, weapon recoil mechanics, and infinite enemy spawning.
17:54 Flight Combat Simulation: Both models generated functional 3D flight logic. The 31B model included ammunition tracers and distinct health metrics, while the local 26B model implemented superior crash/respawn logic and detailed terrain.
The 26B MoE outperformed the dense model in translating a hand-drawn wireframe into a portfolio, creating a "live inference simulation" with unique animation loops.
The 31B Dense included a "sentiment engine" but was subjectively less aesthetically polished than the MoE result.
25:43 Creative Writing & Vision: Tested using a historic novel cover prompt. Both models demonstrated emergent behavior by assigning similar chapter titles ("Cracks in the Porcelain") and interpreting complex domestic/psychological themes from the same image.
32:22 Visual Component Identification: Both models struggled with granular hardware identification. Neither could specifically name a 28BYJ stepper motor or a ULN2003 driver from a schematic, providing generic "DC motor" descriptions instead.
26B MoE: Captured high information density from a complex website reference photo, correctly identifying specific names and executive titles (e.g., CTO Sarah Chen).
31B Dense: Produced a visually superior hero section but suffered from broken image links in the data visualization sections.
40:09 Final Assessment: The 26B MoE (4B active) is the standout for local practitioners, offering a superior balance of speed (22+ TPS at Q8) and reasoning. The 31B Dense model, while capable, currently suffers from low inference speeds (approx. 5 TPS) on available cloud providers and local stability issues.