This video introduces the Mistral Devstrol Small-2507, a newly updated and coding-specific large language model, marking the first Mistral model tested on the channel. The presenter details its key features, including an Apache 2.0 license, a 128k context window, and a design for lightweight agentic coding tasks on consumer hardware like a 24GB GPU. Testing was conducted in two main environments: LM Studio for generic HTML and Python tasks, and VS Code with Klein for more complex agentic game development.
In LM Studio, the model successfully generated a multi-page HTML website, though with a basic UI, and a functional Python-based synthwave game, despite requiring a user-provided image and exhibiting a crude exit. When tasked with developing a 2D side-scroller game ("Mega Slap Slap 4000") in VS Code using Klein, the model demonstrated an ability to establish project structure and self-correct minor errors. However, both initial and refined versions of the game suffered from significant functional issues, including non-moving characters and failed image loading. The presenter attributes these limitations primarily to the constrained 32,768 context length and Q4KM quantization used for testing, suggesting the model's full potential for agentic coding would be realized with higher quality quants and its advertised 128k context.
Summarizing the Mistral Devstrol Small-2507 Model
0:11 Introducing Devstrol Small-2507: The video features the first Mistral AI model tested on the channel, a new coding-specific model (Devstrol Small-2507), an updated version of a previously existing model.
0:55 Key Model Features: It is licensed under Apache 2.0, designed for agentic coding tasks, and is lightweight, capable of running on a single 24 GB VRAM GPU (like an RTX 4090) or a 32 GB RAM Apple Silicon machine.
1:13 Context Window and Benchmarks: The model boasts a 128k context window and scores significantly higher than many other models on included benchmarks, indicating strong hypothetical coding capabilities.
2:51 Testing Environment: The model was tested using LM Studio (with Q4KM quantization and a 32,768 context length, utilizing about 20 GB of VRAM on an RTX 5090 mobile) and in VS Code with Klein for agentic coding.
3:33 LM Studio - Steve's PC Website Test: The model generated a multi-page HTML website that, while aesthetically poor, demonstrated proper multi-page structure and functional hyperlinks.
5:46 LM Studio - Python Synthwave Game Test: It successfully generated a Python-based obstacle avoidance game using Pygame, but required the user to provide a background image and had a non-graceful exit upon losing.
7:47 VS Code with Klein - Game Development Task: The model was tasked with creating a 2D side-scroller game "Mega Slap Slap 4000," including player movement, enemies, and a shooting mechanism.
8:48 Successful Project Setup: The model successfully created the basic project structure, including directories and core HTML, CSS, and JavaScript files for the game.
10:03 Self-Correction and Readme: It demonstrated the ability to identify and fix errors in its own code (e.g., game loop structure, TypeScript errors) and generated a README file with instructions on how to run the game.
10:58 Initial Game Functionality Issues: The first attempt at the game resulted in a non-functional product, with unresponsive character movement and basic cube graphics instead of intended elements.
11:34 Attempted Refinements and Troubleshooting: The model impressively tried to self-correct and create placeholder images for player/alien sprites, even though it ultimately generated empty files for them.
13:06 Continued Game Functionality Problems: Even after refinements, the game remained largely non-functional, with the player unable to move, images failing to load, and the timer not iterating properly.
14:10 Hardware and Quantization Limitations: The presenter concluded that the model's struggles in the agentic coding task were likely due to the limited 32,768 context length used (compared to its advertised 128k) and the Q4KM quantization, which is more noticeable for specific coding models.
15:36 Model's Potential: Despite the observed limitations, the model shows significant potential for agentic coding tasks, especially given its size, suggesting it would be very impressive with a higher context length and less quantization.
16:01 First Mistral Model Tested: This video marks the first time the presenter has tested a Mistral AI model on the channel.