Waymo unveils DeepMind-powered world simulation model

Waymo has revealed its new generative simulation model built on Google DeepMind’s Genie 3, capable of training its self-driving system on a range of edge scenarios including tornadoes, flooded streets—even an elephant on the road. The Waymo World Model leverages Genie 3’s broad world knowledge, acquired through pre-training on a vast and diverse video dataset, allowing the company to explore situations never directly observed by its fleet.

The system generates both camera and LiDAR data through specialised post-training that translates 2D video data into 3D LiDAR outputs tailored specifically to Waymo’s proprietary hardware. Cameras provide the necessary contextualising visual information while the simulated LiDAR input attempts to deliver precise depth information as a complementary signal. 

Waymo’s new model offers three control mechanisms allowing engineers to test counterfactual scenarios through driving action control, adjust road layouts and traffic conditions via scene layout control, and use text prompts to generate different times of day, weather conditions or entirely synthetic scenes. 

The system can also convert ordinary dashcam or mobile phone videos into multimodal simulations showing how the Waymo Driver—the autonomous driving system—would perceive a scene with its sensors. A leaner version of the model achieves dramatic reduction in compute requirements for scenarios that take longer to play out, enabling large-scale simulations.

Of course, Waymo is far from the first company to use a world simulation model to generate training data from autonomous driving. Several companies have been working in this vein for several years already; Wayve’s GAIA-1 multimodal model—arguably the trailblazer for the industry—takes video, text and driving actions as input to generate realistic driving scenes. The company has attracted the interest of Nvidia, winning hundreds of millions from the tech giant in recent investment rounds, as well as praise from Chief Executive Jensen Huang. 

Other firms include Waabi, which uses a system called Copilot4D that leverages a closed-loop neural simulator to create reactive environments where the AI driver and virtual world influence each other in real-time. Nvidia itself provides Cosmos world foundation models including transfer and prediction components to partners including Plus and Oxa, while Pony.ai recently partnered with Moore Threads to scale its PonyWorld generative capabilities.

Waymo, which is owned by Google parent Alphabet, argues that simulating rare scenarios better prepares its autonomous driving system for complex or otherwise unlikely situations, although the company has not shared benchmark results or independent evaluations. The Waymo Driver has logged nearly 200 million fully autonomous miles to date, but racks up billions of miles in virtual worlds before facing scenarios on public roads. 

The announcement comes as Waymo faces safety probes from US authorities following software mishaps in recent months. The National Highway Traffic Safety Administration and National Transportation Safety Board are both investigating a series of incidents involving Waymo robotaxis illegally passing stopped school buses, as well as a separate incident involving a robotaxi striking a child during school drop-off hours. The firm has rejected demands from school districts not to operate its technology in school neighbourhoods.

Waymo plans to deploy commercial robotaxi services in as many as 15 cities by the end of the year, including its first-ever overseas expansions in London. The firm is also testing its technology on Tokyo roads, and has recently confirmed that talks have taken place with Australian government officials about a potential market expansion.