Vayu Robotics (Palo Alto, CA, USA) has released a mobile robot that uses plenoptic sensing and AI—but not LiDAR—to nagivate its surroundings.
The robot, Vayu One, is built to deliver goods from a fulfillment center to consumers.
“The unique set of technologies we have developed at Vayu have allowed us to solve problems that have plagued delivery robots over the past decade, and finally create a solution that can actually be deployed at scale and enable the cheap transport of goods everywhere,” says Vayu Robotics CEO Anand Gopalan.
How? One key technology is plenoptic sensing, which is a type of passive sensing that not only captures information about the intensity of light but other dimensions of light as well.
“The idea of plenoptic cameras is that there is all of this active information about the world in the passive light field that you can extract—except it’s not extractable by standard cameras because they’re only measuring one property of light, which is the average intensity,” explains Gopalan.
He says the company’s plenoptic sensing technology also measures the angle of incidence of the photon (or the angle at which a photon hits a camera’s imaging plane) and the polarization of the photon (or the orientation of the photon). These measurements provide additional information about the shape, texture, and other features of objects and living things in the environment.
In addition to plenoptic sensing, the robotic system also incorporates data from GPS and IMU (inertial measurement unit) technologies.
How the AI Model Works
The visual information captured via the sensor is analyzed using a transformer-based foundation model.
In natural language processing, transformer-based algorithms predict the next word based on the words that have proceeded it, Gopalan explains. Vayu Robotics applies the same approach to visual information, meaning the algorithms predict the next visual “token” based on previous images. “I can predict what the world is going to look like a second from now and two seconds from now and three seconds from now,” he says, noting that this is how humans work.
Meanwhile, foundation models are generalized neural networks that have already been trained on many tasks, such as natural language processing. They provide a starting point for developers who want to create new models without starting from scratch.
Related: Polarization 101: Concepts, Techniques and Technologies
With this approach, pre-mapping of the environment isn’t necessary but training is essential. The models not only need to learn how to predict but also react if the next visual token doesn’t conform to the predictions. “When the prediction doesn’t agree with reality—like somebody jumps in front of your bicycle—you have to take action and do something differently,” he says.
Vayu Robotics trained the neural network model on “millions of miles in a simulated world” in which it confronted countless situations, such as less ideal weather, low light, objects out of place, and people acting in unpredictable ways, Gopalan explains.
Embedded Vision
The AI-enabled models and sensor are combined in a camera, called Vayu Sense, which is manufactured for Vayu Robotics by e-con Systems (Riverside, CA, USA), an OEM camera manufacturer.
The camera incorporates plenoptic technology into a standard CMOS imager and uses proprietary filtering technology. Speaking about the filter, Gopalan says, “We are not saying exactly where in the optical path it is, but it is in the optical path.”
While plenoptic sensing can be expensive, Vayu Robotics’ cameras are similar in price to their RGB counterparts, he says.
Commercial Applications of the Vision-Enabled Mobile Robot
Already, the company has signed a contract with a large unnamed e-commerce organization to deploy 2,500 robots that will deliver merchandise from fulfillment centers to consumers.
Earlier this year, the e-commerce customer piloted the service in several cities in the San Franscisco Bay Area. “We will be expanding those operations further in the coming months to a much larger number of robots,” Gopalan says.
Related: Zalando Incorporating Mobile Robots into Logistics Applications
Vayu Robotics also envisions expanding into other parts of the supply chain. “We have basically built our first form factor robot, but our technologies are applicable across a variety of different types of robots,” he explains.
The company closed a $12.7-million seed funding round in October 2023 with contributions from Khosla Ventures (Menlo Park, CA, USA) and Lockheed Martin Ventures (Bethesda, MD, USA).
In addition to Gopalan, who took Velodyne LiDAR (San Jose, CA, USA) public in 2020, the founders include Mahesh Krishnamurthi, formerly with Apple (Cupertino, CA, USA) Special Project Group and Lyft (San Francisco, CA, USA), and Nitish Srivastava, also from Apple SPG and Geoffrey Hinton’s AI lab at the University of Toronto (Toronto, Canada).
About the Author
Linda Wilson
Editor in Chief
Linda Wilson joined the team at Vision Systems Design in 2022. She has more than 25 years of experience in B2B publishing and has written for numerous publications, including Modern Healthcare, InformationWeek, Computerworld, Health Data Management, and many others. Before joining VSD, she was the senior editor at Medical Laboratory Observer, a sister publication to VSD.