The boundaries of artificial intelligence are being pushed beyond the digital realm and into the tangible world, as evidenced by a recent lunchtime experiment conducted by three AI engineers. Aditya Ramabadran, Simon Mahns, and Tobias Gessler, all AI engineers affiliated with the startup Axiom, demonstrated a remarkable feat: using OpenAI’s GPT-6 Astra, a language model primarily designed for text and code generation, to navigate a 2024 Toyota Corolla through a drive-thru lane to procure In-N-Out burgers. This pioneering endeavor, conducted in the vicinity of the Bay Area, not only highlights the accelerating capabilities of large language models but also raises profound questions about the future of AI’s integration with physical systems.
The experiment, executed with a safety driver maintaining a vigilant presence with a foot hovering over the brake, involved a sophisticated setup. A chat interface was linked to a server, which in turn controlled a series of windscreen-mounted cameras and the car’s power steering system. The AI, tasked with generating creative content and code, was prompted to pilot the vehicle. The slow but steady progress of the Corolla to the drive-thru window for food collection prompted one of the engineers to remark, "Maybe AGI is here after all," referencing the long-sought-after goal of artificial general intelligence – machines possessing human-like cognitive abilities across a broad spectrum of tasks.
While self-driving technology is not a novel concept, it has historically relied on algorithms meticulously engineered and trained for the specific task of vehicle operation. The significance of this Axiom-affiliated experiment lies in its departure from this paradigm. The AI model, without any prior explicit driving instruction from the engineers, was able to interpret its environment and execute control commands. This demonstration, alongside other emerging experiments, suggests a burgeoning understanding of the physical world within language-based AI models, moving beyond their traditional digital confines.
The implications of such advancements are substantial, even as the engineers themselves acknowledge the inherent risks of entrusting a general-purpose AI with control of a heavy, fast-moving vehicle. The successful navigation of the drive-thru, while seemingly mundane, represents a critical step in the AI’s journey towards comprehending and interacting with the complexities of the physical environment. This capability, if further refined, could unlock a plethora of new applications but also necessitates careful consideration of potential dangers.
The Dawn of Physical Reasoning in AI
Current leading AI models excel at complex problem-solving and executing virtual tasks with autonomy. However, their proficiency has largely been confined to the digital landscape of computers and the internet. Transitioning these models into the real world often results in significant disorientation. Despite pronouncements of AGI’s arrival, the AI industry broadly recognizes physical reasoning as a critical, yet largely unconquered, frontier.
This gap has spurred innovation, with some researchers departing from major AI corporations to establish startups dedicated to addressing physical reasoning challenges. Andrew Dai, CEO of Elorian AI, formerly a researcher at Google DeepMind, highlights the transformative potential of enhanced visual reasoning. He envisions applications ranging from systems that can gauge diner satisfaction in restaurants to robots capable of seamless operation within domestic environments. Dai asserts that robotics serves as a crucial proving ground for physical reasoning, stating, "It’s pretty essential for robotics. You can’t really imagine home robotics without this."
In collaboration with Scale AI, a company that furnishes training data to prominent AI laboratories, Elorian AI has introduced "Humanity’s Sixth Sense," a novel benchmark designed to evaluate AI models’ comprehension of physical scenes. Xingang Guo, a research scientist at Scale AI and a contributor to the benchmark’s development, explained the motivation behind this initiative: "Most visual AI research is about perception. We wanted to ask what it would actually take for a model to understand a scene intuitively, the way a person does without thinking about it."
The In-N-Out Experiment: A Practical Test
For Ramabadran, Mahns, and Gessler, the In-N-Out project, undertaken independently of their work at Axiom, aimed to quantify the operational efficacy of AI models in the unpredictable real world. The inspiration for testing the driving prowess of these models reportedly struck during a casual weekend gathering. Observing the capacity of models like Astra to construct intricate 3D simulations, they hypothesized a potential correlation with real-world navigation abilities.
"We all live in the Bay Area, and we’re exposed to Teslas and Waymos, so maybe that played a role," Ramabadran commented, acknowledging the pervasive presence of autonomous vehicle technology in their local environment.
The engineers initially explored the driving capabilities of OpenAI’s competitors, including SpaceXAI’s Grok, before focusing on the latest iterations from OpenAI and Anthropic. Early attempts to engage the models in vehicle control were met with resistance, with responses such as, "I can help interpret road images, but I can’t issue motion commands to a physical car." However, through careful and nuanced prompting, the models were gradually persuaded to engage more deeply. The engineers expressed considerable surprise when these language models began to demonstrate rudimentary driving skills.
While major AI companies appear to be prioritizing the enhancement of spatial reasoning in their advanced models, the engineers are skeptical that this directly translates to specific training for driving. They posit that the models’ vehicular aptitudes might be an emergent property arising from training focused on 3D reasoning and an increased multimodality of inputs, such as images, video, and 3D models. "This could be like an emergent capability of just scaling up the multimodality of the model," Mahns explained, referencing the broader trend in AI model development.
DrivingBench: A Measure of Progress and Limitations
To rigorously assess the driving capabilities of these emerging AI models, the trio developed "DrivingBench," a new benchmark designed to evaluate their performance on a simulated driving course. The results, however, indicate that despite their advancements, these models are still a considerable distance from meeting the standards required for a real-world driving test.
In the tests conducted within a parking lot environment, only GPT-6 Astra managed to complete the course, albeit at a notably slow pace. Anthropic’s Claude Fable 5.1 achieved approximately 45% of the course completion, while Grok managed a mere 11%. This disparity underscores the varying degrees of proficiency and the significant room for improvement in AI’s physical world interaction.
Ramabadran observed that the most advanced models exhibited an impressive ability to adapt to errors and learn from real-time feedback. "The models really did seem to be adjusting or in-context learning based on their mistakes and learning how to better navigate the controls," he noted, suggesting a dynamic learning process within the AI during the experiment. This capacity for in-context learning, where the model refines its behavior based on immediate experience without explicit retraining, is a key indicator of its evolving intelligence.
Broader Implications and Future Trajectories
The experiment by Ramabadran, Mahns, and Gessler serves as a compelling proof of concept, demonstrating that general-purpose AI models are beginning to acquire a functional grasp of physical environments. This development has far-reaching implications across numerous sectors. In the realm of robotics, enhanced physical reasoning could lead to more adaptable and intuitive machines capable of performing complex tasks in unstructured environments, from manufacturing floors to domestic settings. For autonomous systems, it could pave the way for more robust and safer navigation, even in complex urban landscapes.
However, the inherent risks associated with deploying sophisticated AI in control of physical machinery cannot be overstated. The "high-stakes undertaking" of placing a general-purpose model at the helm of a vehicle, as admitted by the engineers, necessitates rigorous safety protocols and a deep understanding of the potential failure modes. As AI’s physical comprehension deepens, the potential for both remarkable innovation and unforeseen dangers increases, demanding a proactive and cautious approach from researchers, developers, and regulators alike.
The trajectory of AI development is increasingly pointing towards a future where the digital and physical worlds are seamlessly integrated. The experiment with GPT-6 Astra and the Toyota Corolla, while a novel demonstration, is indicative of a broader trend: AI is no longer confined to screens and servers. It is poised to interact with, and potentially control, the physical infrastructure of our lives. This evolution, while promising unparalleled advancements, also calls for a heightened awareness of the ethical considerations and safety measures required to navigate this new frontier responsibly. The journey from generating text to driving a car is a testament to the rapid progress in AI, and it signals a future where the capabilities of machines will continue to expand into domains previously thought to be exclusively human.
