This video presents a head-to-head coding challenge between two leading large language models (LLMs), GPT-5 Pro and Grok-4 Heavy, to generate a 3D Python game titled "Moon Dirt Bike Simulator," later re-attempted in HTML, JavaScript, and CSS. The speaker, having paid for Grok-4 Heavy, aims to assess their code generation capabilities for a moderately complex task, including dynamic gravity, free-roam, and a visual representation of Earth. Despite multiple attempts and prompting for fixes, both models largely failed to produce a functional or playable game, exhibiting numerous errors, incomplete implementations, and visual glitches. The Grok-4 Heavy model managed to create a minimally functional Python game (though still obscure and unplayable as intended), while GPT-5 Pro consistently failed to produce working Python code and generated a "disturbing" visual in the web version. The speaker expresses significant disappointment, noting that these results contradict his extensive positive experiences with AI in coding and suggests that current browser-based LLM performance may not reflect their true potential, which is better realized in "agentic coding frameworks."
Summarizing the LLM Code Generation Battle: GPT-5 Pro vs. Grok-4 Heavy
0:11 Introduction to the Test: The speaker sets out to test GPT-5 Pro against Grok-4 Heavy, two "most performant LLMs," in a code generation challenge. He paid $300 for Grok-4 Heavy for this purpose.
1:11 Challenge Description (Python): The LLMs are tasked with creating a "Moon Dirt Bike Simulator" game in Python, featuring 3D graphics, a moon environment with Earth visible in the sky, free-roaming, and a gravity slider to adjust between Earth and Moon gravity.
2:33 Initial Generation & First Impressions: GPT-5 Pro is started first, expected to take about 15 minutes. The speaker notes Grok's aesthetically pleasing "agents working" UI and timer.
4:14 First Python Results: After 5 minutes, Grok-4 Heavy finishes; GPT-5 Pro finishes at 9 minutes 25 seconds. Both results are described as "disastrous." GPT-5 Pro's script fails to open, while Grok's opens but is not functional ("might as well not do anything").
8:11 Attempting Fixes: The speaker provides the error messages to each model, expressing "disappointment" to prompt better results.
9:21 Regeneration & Stubbornness: GPT-5 Pro responds quicker but only provides a fix, not a full regenerated script, which the speaker finds frustrating due to models being "stubborn against regenerating full scripts." Grok-4 Heavy regenerates the entire script, taking longer.
11:37 Second Python Test: Grok's second attempt opens and allows some movement and camera control (right-click and WASD), with the Earth visible, but the bike is largely unplayable and floats. GPT-5 Pro's second attempt still fails to work.
14:50 Third Round of Fixes: Both models are given a third chance to improve their scripts.
15:33 Third Python Test: GPT-5 Pro's third attempt still fails to work, leading the speaker to question if something is wrong with the model at that moment and express deep "dumbfounded" disappointment. Grok's third attempt shows the bike moving, steering (front wheels turn), and a partially working gravity slider, though the physics remain obscure and the game is still not fully playable.
20:23 Shifting to Web Technologies (HTML/JS/CSS): Due to Python failures, the prompt is changed to generate the same game using HTML, JavaScript, and CSS, hoping for better performance in a "more performant" area for LLMs.
21:23 First Web Results: GPT-5 Pro's web attempt is described as "disturbing," featuring a "smiley face" and a cube between cylinders, with non-functional movement. Grok's web attempt is "almost the same exact result" as its Python one but still non-functional.
26:17 Discovering Grok's Voice Mode: The speaker unexpectedly discovers and praises Grok's voice interaction mode, calling it "quite nice to use" and "very good."
26:45 Cross-Model Fixing: Grok is asked to fix GPT-5 Pro's HTML/JS/CSS code, and GPT-5 Pro is asked to fix Grok's HTML/JS/CSS code.
27:08 Thinking Times: Grok-4 Heavy consistently takes longer to "think" than GPT-5 Pro.
GPT-fixed Grok's code: This result is deemed "better than it was by a long shot," with working gravity, a visible map, and subtle wheel movement, though still "obscure" and not a polished game.
Grok-fixed GPT's code: This attempt still fails to run.
29:09 Conclusion & Reflection: The speaker concludes the video with a "troublesome result," reiterating that this experience is "not reflective" of his usual positive AI coding experiences (e.g., Arduino). He suggests that the models' poor performance in a direct web environment indicates they are "poised towards coding in agentic frameworks," where they can iterate and fix errors without constant full script regeneration.
30:56 Final Thoughts on Grok: Despite the overall disappointing coding results, the speaker praises Grok's UI and its voice mode, promising a more in-depth review of Grok due to its high cost.