This lecture by Lanxiang Hu from UCSD introduces a novel framework for evaluating and training Large Language Models (LLMs) using computer games, presented as the "LM Game Bench." The talk argues that traditional static benchmarks are vulnerable to data contamination, while games offer dynamic, interpretable, and scalable evaluation environments. Hu demonstrates that top-tier LLMs initially struggle with simple games like Super Mario, revealing challenges in perception, latency, and memory. To address this, his team developed a modular "gaming harness" with perception, memory, and reasoning components, which significantly improves model performance. The research explores various game genres, from puzzle games like Sokoban (testing long-horizon planning) to text-heavy games like Ace Attorney, noting the need to mitigate data contamination in the latter. The talk presents a public leaderboard, analyzes how gaming performance correlates with core abilities like math and coding, and details successful experiments in training models on games, which show promising generalization to other games and agentic tasks.
Evaluating and Training LLMs on a Game Arena
0:53 - Why Use Games for Evaluation?: Games provide a rich, scalable, and intuitive environment for AI evaluation. Unlike static benchmarks, classic games have well-defined states, actions, and verifiable rewards, making them ideal for assessing a model's planning and reasoning capabilities.
1:45 - LLMs Struggle Out-of-the-Box: A demo shows that even state-of-the-art models like GPT-4o and Claude 3.7 fail at basic tasks in Super Mario. This highlights a significant gap between their abstract reasoning abilities and their capacity to act effectively in dynamic environments.
7:07 - Strategic Game Selection: The benchmark uses a diverse set of popular games to test different skills:
Puzzle/Board Games (Sokoban, 2048, Tetris): Test long-horizon planning and spatial reasoning.
Real-time Games (Super Mario Bros.): Test reaction time and physical understanding.
Visual Novels (Ace Attorney): Test text comprehension and reasoning in data-rich scenarios.
11:14 - The "Gaming Harness" Solution: To overcome LLMs' inherent weaknesses (e.g., poor vision, high latency, no memory), a modular "gaming harness" is introduced as an agentic workflow. It includes:
Vision Perception Module: Converts game images into structured text.
Memory Module: Provides gameplay history to enable long-term planning and prevent repetitive mistakes.
Reasoning Module: Synthesizes information to decide on the next action.
17:10 - The "Knowing and Doing" Gap: In real-time games, models often generate a sensible high-level plan (e.g., "jump over the pipe") but fail to translate it into the precise, low-level actions required for execution (e.g., "press jump key for 10 frames").
19:17 - Mitigating Data Contamination: In text-heavy games like Ace Attorney, models often memorize the plot from pre-training data. To ensure fair evaluation, researchers use mitigation techniques like replacing character names with generic IDs and rewriting game text.
24:52 - Public Leaderboard and Rankings: The project features a public leaderboard to rank models based on their gaming performance, both with and without the gaming harness. The results indicate that Claude 3 series models currently lead in performance.
28:42 - Correlating Gaming with Core Abilities: Analysis reveals that gaming performance is a composite of other fundamental skills. Performance in games like Sokoban and Tetris strongly correlates with math and coding benchmarks, while performance in Super Mario correlates with physics and spatial reasoning benchmarks.
35:08 - Generalization from Game-Based Training: An experiment training a 7-billion parameter model on Sokoban showed impressive results. The model not only improved on the trained game but also generalized its learned skills to unseen games (Tetris) and other complex agentic tasks (Webshop).
41:29 - Pokémon Red Considered a Poor Benchmark: The speaker argues that using Pokémon Red for evaluation is flawed. Its tasks are either too easy (battle control), too hard without assistance (navigation), or prohibitively expensive and slow for long-term planning, often requiring "cheats" like grid overlays that undermine the evaluation.
43:33 - The Future of Game Evaluation: The talk suggests a shift towards evaluating higher-level capabilities. Instead of executing low-level actions, LLMs could be evaluated on their ability to call specialized tools (e.g., an A* pathfinding algorithm), creating more efficient and meaningful assessments.