Expert Reviewer Group: AI Safety & Autonomous Systems Engineering Leadership

Abstract: The Waymo World Model

The Waymo World Model (WWM) is introduced as a frontier generative model designed for large-scale, hyper-realistic autonomous driving simulation, functioning as a critical component of Waymo’s AI safety and validation ecosystem. The WWM is an adaptation of Google DeepMind’s Genie 3, leveraging its extensive world knowledge—derived from diverse video pre-training—to simulate exceedingly rare ("long-tail") and safety-critical scenarios that are difficult to capture in real-world fleet data. Key technical features include high-fidelity, multi-modal sensor generation, providing both camera and Lidar data outputs. The model emphasizes high controllability through three mechanisms: driving action input, scene layout configuration, and flexible natural language prompts. This capability allows engineers to rigorously test the Waymo Driver against extreme weather, natural disasters, and complex counterfactual events, significantly enhancing the safety and scalability of the autonomous system.

The Waymo World Model: Key Technical Capabilities and Simulation Rigor

• Foundation and Purpose: The Waymo World Model is built upon Google DeepMind's Genie 3 general-purpose world model, adapted specifically for the demanding rigor of the driving domain. It is responsible for generating hyper-realistic simulated environments, complementing nearly 200 million real-world fully autonomous miles traveled by the Waymo Driver.
• Multi-Modal Realism: Unlike models trained solely on proprietary road data, the WWM transfers Genie 3’s 2D video knowledge into 3D Lidar and camera outputs, ensuring complementary and precise depth information vital for autonomous sensor suites.
• Handling Long-Tail Scenarios: By utilizing Genie 3's vast pre-training data, the model can simulate events virtually impossible to capture at scale in reality (e.g., tornadoes, floods, lions, or vehicles driving recklessly off-road), creating a more rigorous safety benchmark.
• Strong Controllability Mechanisms: The WWM offers fine-grained control over simulations via three primary input modalities:
  • Driving Action Control: Enables responsive simulation adherence to specific driving inputs, supporting the creation of "what if" counterfactual events to test driver confidence and yielding behavior.
  • Scene Layout Control: Permits customization of road layouts, traffic signal states, and the targeted placement and behavior of other road users.
  • Language Control: Provides the most flexible tool for world mutation, allowing modifications via simple language prompts for parameters such as time-of-day (Dawn, Noon, Night) and weather conditions (Foggy, Snowy, Rainy).
• Conversion of External Footage: A unique capability of the WWM is its ability to convert standard dashcam or mobile phone videos into a multi-modal simulation, showing how the Waymo Driver’s sensors (camera and Lidar) would perceive that exact scene, ensuring the highest degree of realism and factuality.
• Scalable Inference: An efficient variant of the WWM has been developed to simulate longer, more complex scenes (e.g., negotiating narrow lanes or freeway traffic) while significantly reducing computational requirements and maintaining high fidelity.
• Safety Implication: By simulating the "impossible," the WWM proactively prepares the Waymo Driver for the most rare and complex edge-case scenarios, moving beyond purely reconstructive simulation methods that suffer visual breakdowns when routes deviate significantly from original driving data.