John Carmack, detailing his work at Keen Technologies, outlines his current focus on Reinforcement Learning (RL) that prioritizes continuous, interactive experience, contrasting it with the prevailing Large Language Model (LLM) pre-training paradigm. He posits that foundational human-like intelligence requires online learning in dynamic, real-world-like environments. Using Atari games as a benchmark, Carmack's team is developing methods to overcome critical challenges in real-time RL, including speed, latency, catastrophic forgetting, and effective transfer learning. A notable aspect of their research involves a physical Atari setup where an AI agent learns to play games by observing a screen and manipulating a joystick, confronting real-world robotics challenges. The talk delves into the technical limitations of current RL algorithms and proposes areas for architectural and training improvements to achieve more robust and generalizable AI, emphasizing the need for agents to learn quickly, retain knowledge, and apply it to novel situations.
Summarizing the Talk Notes: John Carmack on AI/RL Research at Keen Technologies
0:00 Background & Motivation: John Carmack (founder of Id Software, Armadillo Aerospace, Oculus) details his shift to AI research at Keen Technologies, driven by the belief that learning from interactive experience, akin to humans and animals, is "The Most Important Thing." He explicitly chose not to focus on LLMs, viewing interactive learning as fundamentally different from their "throw-everything-in-a-blender" pre-training.
1:45 Initial Thoughts & DeepMind's Atari Work: Carmack's background biases him towards virtual environments and games. He acknowledges DeepMind's 2013 Atari work but highlights significant caveats: models needed 200 million to billions of frames (over a month of 24/7 play) to approach human baselines, and fundamental aspects of continuous, efficient, lifelong, multi-task online learning remain unsolved.
7:00 Missteps & Refining Focus: Early missteps included starting too low-level (C++/CUDA instead of PyTorch), avoiding larger experiments, and choosing Sega Master System games over Atari. He settled on Atari for its established research body and decided to defer passive video understanding to prioritize fundamental multi-game play challenges.
9:00 Atari as a Research Platform: Carmack argues Atari is still valuable for open research due to its unbiased and diverse gameplay, revealing subtle issues like sprite flicker and deterministic exploitation. He notes that even "solved" Atari results from modern models like BBF and Efficient Zero have caveats regarding noisiness and evaluation.
13:00 Bridging to Reality: Not a Turn-Based Game: A core challenge is that the real world is continuous, unlike turn-based game environments. Training in a single environment (rather than parallel, which is easier) is crucial for biological intelligence.
14:30 Real-time RL Challenges:
Speed: Agent performance is limited by how often its policy can be evaluated, requiring efficient implementations like CUDA graphs and overlapping training.
Latency: Actions in the real world have delayed effects, causing many modern high-performance RL algorithms (which assume immediate action-effect) to fail. Carmack suggests testing algorithms with a "delay line" for actions.
16:30 Physical Atari Setup: Carmack is open-sourcing a physical Atari 2600+ setup, using a camera to observe the screen and servo motors to manipulate the joystick. This tackles real-world robotics problems, including sensory input, latency, mechanical reliability (e.g., servo/joystick failures), and surprisingly, the challenge of reliably reading game scores.
22:00 Rewards, Exploration & Meta-Curiosity: Real-world tasks often have sparse rewards, unlike the dense rewards RL algorithms prefer. He discusses "hard exploration games" and proposes intrinsic rewards and "meta-curiosity" (e.g., an agent choosing which game to play based on its own curiosity and performance improvement) to drive learning.
25:00 Catastrophic Forgetting & Transfer Learning: A major hurdle is that neural networks suffer from "catastrophic forgetting," losing old skills when learning new ones sequentially, unlike humans who retain knowledge over long periods. Current agents often show "negative transfer," performing worse on new games after learning others. Carmack proposes a new benchmark that cycles through games to measure learning speed, forgetting, and transfer.
30:00 Architectural & Theoretical Considerations: Carmack delves into the underlying function approximation of neural networks, noting issues like every weight update changing all outputs and the surprising underperformance of large image models for RL. He discusses exploring new ConvNet architectures (e.g., isotropic, dilated, semi-dense CNNs) and value representations to improve generalization and learning stability, favoring recurrence over frame stacks for state representation, aligning with biological brains.