Amazon Develops Computer Vision Solution to Assist Delivery Drivers

Amazon is rolling out a computer vision solution that automatically identifies packages to be delivered at each stop, saving drivers time and effort.

With the solution, Vision-Assisted Package Retrieval (VAPR), drivers no longer need to organize packages by stops, read labels or other identifying information, or scan barcodes manually to ensure they have the correct package(s) for a given address. Instead, when a driver arrives at a delivery location, VAPR will project a green “O” on all packages to be delivered at that stop and a red “X” on all other packages, according to a news release from Amazon (Seattle, WA, USA).

A driver waits to enter the van’s cargo area at each stop until the VAPR system lets him or her know that it has found the right packages for the stop. The technology alerts the driver with both a beep and a green circle that appears on the navigation screen, which is mounted in the front of the van. By looking at the screen, the driver also can see how many packages are slated for delivery at a given stop.

Amazon estimates that the new technology will save drivers about 2-5 minutes per stop, totaling 30 minutes per route, based on feedback from drivers who tested the technology. By early 2025, the company plans to roll out VAPR in 1,000 of the company’s new electric delivery vans. The company has more than 390,000 drivers operating more than 100,000 vans to deliver millions of packages daily.

VAPR uses Amazon Robotics Identification (AR-ID), a scanning capability involving computer vision and machine learning, which was originally developed for fulfillment centers to replace manual barcode scanning.

With AR-ID in the fulfillment centers, Amazon says a freestanding camera system that runs at 120 fps automatically scans barcode labels as packages are handled by an employee working nearby. This process allows employees to handle packages with two hands, rather than holding a package in one hand and a barcode scanner in the other hand.

Related: Amazon is Testing an Autonomous and Collaborative Robot

For the in-van application, AI-ID can locate and decipher multiple barcodes in real time as it processes its surroundings.

How Amazon's Vision-Assisted Package Retreival Works

Amazon’s developers trained machine learning models to recognize different labels and packages in a variety of lighting conditions specifically in an in-van environment. Those models are incorporated into VAPR, Amazon says in the press release.

The technology system uses specially designed automotive light projectors and cameras. The software and hardware components are integrated into a custom enclosure, which is mounted in the van’s cargo area. The system’s components also are integrated with the van’s navigation system.

David Dechow, founder of Machine Vision Source (Salisbury, NC, USA) and a machine vision expert, says, “a really big achievement, I think, is the interaction between the identification system, the package list in the cloud, and the driver’s route. It seems the system knows what packages need to be delivered at a spot and the software is tied into the van’s navigation system.”

Dechow notes that the automated code reading aspect of the application is not new. In addition, the use of light projectors to aid picking operations is common in the industry, and numerous vendors offer solutions, although “the mechanics of aiming the projection are admittedly complex,” he explains.

Speculating on how the system works, Dechow says the initial layer of processing probably identifies and isolates the label(s) on the package. This reportedly is done using ‘trained machine learning models,’ but it’s unclear exactly what that means technologically. It would be realistic to assume that this segmentation part is deep learning, but other trained recognition models could work, too,” he says.

“The final vision process is to read the barcode(s) on each label. The reading part likely is standard vision algorithms (but could be deep learning as well),” Dechow adds. 

Related: Amazon Testing Robotic Arm that Identifies Individual Products

Amazon says it built the software layers for VAPR using two of its commercially available tools. Developers used SageMaker—a software service to build, train, and deploy machine learning models—and IoT Greengrass—a cloud service that helps developers create software for edge devices.

Inside the van, packages are organized in totes, with each tote holding packages for multiple stops, and drivers work through the totes as they progress on their route, explains Maya Vautier, in Amazon Delivery’s public relations team.  “The number of packages per tote varies, as does the number of totes per route.”

Bobby Garcia, a driver with Bloomfield Logistics (Haverhill, MA, USA), an Amazon Delivery Service Partner, has been testing VAPR. Explaining the impact of the new system on his daily routine, Garcia says, “Before, it could take me anywhere between 2 and 5 minutes to empty a tote and organize packages for the next stops. Now, with VAPR, this whole step just takes me about a minute.”

Other AI Solutions to Help Drivers

Amazon also is implementing other technology improvements to help drivers, according to another news release. For example, the company’s route planning system uses machine learning models that take into consideration such factors as weather and walking distances.

Related: Amazon Plans Major Data Center Expansion in Virginia's Spotsylvania County

The company also uses predictive models to identify safety risks—such as unpaved roads or railroad crossings—on drivers’ routes. By the end of 2024, Amazon expects to identify 200 million risk characteristics and provide 18 million safety cues to drivers through an app for drivers.

“We’re always working on new ways to help DSPs make their drivers’ day-to-day experience safer and simpler,” Beryl Tomay, vice president of transportation services at Amazon, writes in the news release.