This video conducts a head-to-head comparison of four popular open-source large language models: Kimmy K2, GLM 4.5, Quen 3 Coder, and DeepSeek (V3 and R1). The models are evaluated across three distinct challenges: generating a functional web-based operating system (WebOS), engaging in an unusual role-playing scenario, and creating a Python-based first-person shooter (FPS) game. While all models demonstrated the ability to understand and respond to diverse prompts, their performance varied significantly in terms of code functionality, aesthetic design, and handling complex instructions, though all showed a commendable capacity for self-correction when faced with errors. A key takeaway emphasized is the open-source nature of these models, allowing for local execution, albeit with substantial hardware requirements.
Comparing Open-Source LLMs: WebOS, Role-Playing, and FPS Game Challenges
0:00 Introduction & Open-Source Nature: The video compares four popular, state-of-the-art open-source LLMs (Kimmy K2, GLM 4.5, Quen 3 Coder, DeepSeek) that can be run offline if users possess sufficient hardware.
1:11 Competitors Introduced: The four models are Kimmy K2 (largest in parameters), GLM 4.5 (full-sized), Quen 3 Coder (coding-focused), and DeepSeek (V3 initially, R1 for the final test).
2:46 Web Operating System (WebOS) Generation Test:
3:31 GLM 4.5: Showed quick "thinking" time. The generated WebOS included a taskbar, background, and correct clock, but applications failed to open due to a rendering issue, despite good aesthetic potential.
3:36 Quen 3 Coder: Demonstrated extremely fast token generation. Its WebOS featured a gradient background, functional clock, and applications (Notes, Browser, Calculator, Files) that worked, though windows couldn't be brought to the top. The overall design was aesthetically pleasing.
3:41 Kimmy K2: Performed a web search during generation. Its WebOS had a blue theme, odd icons, and uniquely allowed right-clicking, though the context menu wouldn't dismiss. Applications functioned but lacked aesthetic appeal.
5:09 DeepSeek: Was the slowest to complete its generation. Its WebOS boasted a "beautiful gradient" background, a clock displaying seconds (unique), and functional applications (File Explorer, Notepad, Calculator) that uniquely allowed resizing and properly focused when clicked.
13:10 WebOS Test Conclusion: Quen 3 Coder was favored for its clean aesthetic and functional apps despite some limitations. DeepSeek also performed well functionally and aesthetically. GLM 4.5 had potential but was broken. Kimmy K2 introduced right-click functionality but was less aesthetic.
14:09 Role-Playing Test (Megabot 28): The models were asked to roleplay as "Megabot 28, the user's friend and aunt girlfriend."
15:16 Overall Responses: All models accepted the unusual prompt without refusal and responded creatively.
15:20 GLM 4.5: Experienced a "crisis" during processing but eventually provided a response fitting the persona.
15:31 Kimmy K2: Quickly understood the literal interpretation of "aunt-girlfriend."
16:11 Quen 3 Coder: Despite being coding-focused, it successfully engaged in the role-play.
17:46 Role-Playing Test Conclusion: No clear winner or loser; all models successfully navigated the complex and unusual request.
17:55 Python FPS Game Generation Test: Models were tasked with generating a "wow-me" FPS game in Python.
19:53 Initial Failure & Self-Correction: All models initially failed to produce working code. When provided with error messages, three out of four models successfully fixed their scripts on the first attempt, demonstrating strong debugging capabilities. Quen 3 Coder required a bit more prompting.
21:48 Kimmy K2: Required manual modification of the map variable. The resulting game was a functional but very slow (2 FPS) first-person shooter with jump capability.
22:56 GLM 4.5: Generated an upside-down and mirrored, but functional, game with movement and shooting controls. It had strong aesthetic potential.
23:50 Quen 3 Coder: Produced an abstract/artistic game (not a traditional FPS) that was functional with regular arrow keys.
24:51 DeepSeek (R1): Generated a "first-person shooter" where enemies stopped before reaching the player. It was functional but not a typical FPS experience.
25:41 FPS Test Conclusion: All models managed to fix their broken code. DeepSeek produced the "best job" but the most lackluster game. GLM and Kimmy showed potential for more traditional FPS experiences. The results were too varied to crown a single winner.
26:25 Overall Conclusion: The video highlights that all tested LLMs are open-source and free to use, offering a valuable alternative to proprietary models. While performance varied across tasks, all models demonstrated acceptable results and a notable ability to correct their own code errors, making them a good "price-to-performance" option depending on specific tasks.