This video features a presentation by Lanxiang Hu from UCSD on Game Arena, a benchmark designed to evaluate the reasoning and agentic capabilities of Large Language Models (LLMs) through interactive computer games. The talk addresses the shortcomings of existing static and dynamic benchmarks, proposing games as a rich, scalable, and intuitive evaluation environment.
Hu explains that top-tier LLMs still struggle with many games, necessitating a "gaming harness"—a modular agentic workflow with vision, memory, and reasoning components—to overcome limitations in perception, latency, and planning. The presentation details challenges and solutions for different genres, such as the "knowing-doing gap" in real-time games like Super Mario Bros. and data contamination in text-heavy games like Ace Attorney.
The research presents a leaderboard ranking various LLMs, studies the correlation between gaming performance and traditional skills like math and coding, and explores how training on games generalizes to other tasks. Findings show that while the gaming harness significantly boosts performance, generalization to non-gaming tasks like math remains a challenge. The talk concludes by discussing problematic evaluations (e.g., Pokemon Red) and future directions, including hierarchical planning and multi-agent simulations.
Game Arena: A New Frontier for Evaluating LLM Reasoning and Agency
0:03 Introduction: Speaker Lanxiang Hu introduces the project, Game Arena (also referred to as LM Game Bench), which uses computer games to provide a new perspective on AI model evaluation.
1:51 The LLM Performance Gap: A demo of models like GPT-4o and Claude 3 playing Super Mario Bros. shows that even state-of-the-art LLMs perform poorly, often failing immediately. This highlights a significant gap between their reasoning capabilities and their ability to act in dynamic environments.
5:10 Why Use Games for Evaluation?: Games are ideal for evaluation because they offer well-defined states, actions, and verifiable rewards. They are also diverse, scalable, and intuitive, making them a powerful alternative to static benchmarks that suffer from data contamination.
6:51 The "Gaming Harness" Concept: LLMs cannot be placed directly into games due to weak vision, high latency, and a lack of memory. The solution is an agentic workflow called a "gaming harness" designed to scaffold the model's abilities.
11:53 Core Harness Components: The modular gaming harness consists of three main parts to make gameplay feasible:
Vision Perception Module: Converts image-based game states into textual descriptions to prevent the LLM from getting confused.
Memory Module: Records gameplay history to enable long-horizon planning and prevent the model from making repetitive mistakes.
Reasoning Module: Synthesizes information from the other modules to generate an action, often using a chain-of-thought process.
16:14 Challenges in Real-Time Games: Latency-sensitive games like Super Mario present unique challenges. A "knowing-doing gap" is observed where a model can form a sensible plan (e.g., "I need to jump over the pipe") but fails to translate it into the precise actions required (e.g., "press jump for 10 frames").
18:45 Mitigating Data Contamination: In narrative-heavy games like Ace Attorney (referred to as "Asotony"), LLMs often have the plot memorized from their training data. To ensure fair evaluation, the benchmark uses mitigation techniques like replacing character names with generic ones ("entity replacement") and rewriting in-game text.
22:52 Harness Effectiveness: Data shows that applying the gaming harness leads to a statistically significant improvement in model performance across most games, confirming it is essential for a meaningful evaluation.
24:27 The Game Arena Leaderboard: The project hosts a public leaderboard that ranks LLMs across six games, both with and without the agentic harness. At the time of the talk, Claude 3 models were leading the rankings.
29:21 Gaming Performance as a Composite Skill: A linear regression model reveals strong correlations between performance in specific games and established capabilities. For example, performance in Sokoban and Tetris correlates with math and coding skills, while Super Mario relates more to physics and spatial reasoning.
32:59 Training Analysis & Generalization: In an experiment, an LLM trained on the game Sokoban showed improved performance on harder Sokoban levels, generalized to another game (Tetris), and even improved on a separate agentic web task. However, this training did not lead to improvements in math or coding.
41:35 Case Study: The Problem with Pokemon Red: The speaker argues that using Pokemon Red for evaluation is problematic. The tasks are either too easy (battle control), too hard (navigation, which requires an extensive harness), or too costly and time-consuming (long-term team building).
45:35 Q&A - Future of Agentic Gameplay: The Q&A discusses moving toward more human-like, hierarchical planning. Instead of generating low-level actions, an LLM could make a high-level plan (macro) and call a specialized tool, like an A* search algorithm, to execute it (micro).
51:51 Future Directions: The research aims to explore more complex environments, including economic simulations and multi-agent games like Monopoly or Diplomacy, to further probe the strategic and collaborative capabilities of LLMs.