Remi Tachet des Combes

Senior Applied Scientist @ Wayve

Lingo & GAIA: Transforming Autonomous Driving with Large Language Models and Generative AI

In this talk, I will present LINGO and GAIA, two recent models developed at Wayve. LINGO is a pioneering open-loop driving commentator that utilises natural language processing to interpret and articulate driving scenes. Its unique “show and tell” feature employs referential segmentation to visually highlight areas of interest within a scene, enhancing the model’s interaction with its environment. This capability significantly advances autonomous vehicles (AV) by improving accuracy in describing surroundings, addressing model hallucinations, and bolstering safety communication. By integrating vision, language, and action, LINGO represents a critical step towards Vision-Language-Action Models (VLAMs), aiming for human-like communication and trustworthiness in AV technology.

GAIA is an advanced generative world model designed to simulate realistic driving scenarios. It leverages video, text, and action inputs to build representations of the environment and its future dynamics, enhancing AV decision-making and safety. Comprising an image encoder and a 6.5 billion parameter autoregressive transformer trained on extensive driving data, GAIA showcases scalability and superior video generation quality. Its ability to adjust scene features like weather and time, alongside predicting diverse futures and interactions with other agents, positions GAIA as a valuable tool for AV development.