Vision system spots hidden items

Object-recognition software helps identify and bill grocery items in shopping carts.

By C. G. Masi, Contributing Editor

Studies have shown that, on average, grocery stores lose $50,000 per year to items being wheeled out on the bottom of shopping carts without being checked out and paid for. Sometimes clerks forget to ask about bottom-of-basket (BOB) items, and sometimes people forget they are there.

After having to write off $150 million per year in BOB losses, executives at a US grocery-store chain contacted robotics-software developer Evolution Robotics (ER) for help. “They heard about how robotics-system developers had been using our object-recognition software,” recalls Mike McWilliams, vice president of sales at ER, “and wanted to know if it could be used to identify packages on the bottom baskets of shopping carts as they went past the checkout counter.”

Figure 1. The Evolution Robotics LaneHawk vision and lighting hardware fits behind a window in the checkout counter base and recognizes packaged products. It avoids false positives such as a handbag by flagging only items for which it has been trained. The system identifies items by UPC code to the POS checkout equipment, so the clerk can add them to the sale without bringing them to the counter top for scanning.

ER developers were able to design a BOB checking system, the LaneHawk visual scanner, which could retrofit onto existing point-of-sale (POS) systems. The result is that the grocery chain says it has been able to recover 85%-93% of losses in its stores (see Fig. 1).

Visual pattern recognition

The engineers started with their visual pattern-recognition (ViPR) software, which Sony, Honda, Toyota, Samsung, and other companies have put into products. Sony Entertainment Robot Company, for example, has licensed the ViPR technology for its robotic pet dog, AIBO (see Vision Systems Design, July 2005, p. 20).

ViPR is a C++ application designed for robotic manipulation, human-robot interaction, and security. It can be used in machine-vision applications for object identification and eye-hand coordination. The software recognizes planar, textured objects, but can also identify three-dimensional objects composed of planar, textured structures or slightly curved components. Each feature can be uniquely described by the texture in a small window of pixels around its location. The software creates a model of the object as the coordinates of all these features and their texture descriptions.

To recognize an object in a new image, the algorithm collects features from the image, then attempts to associate these features with features in its database of model objects based on similarities between their texture descriptions. Whenever database-object features score high on similarities with new-image features, the algorithm marks that object as a possible match candidate.

The algorithm attempts to refine its search by computing affine transformations that might improve the correlation between the new image and each possible match candidate. Affine transforms involve changes of scale, rotation, and tilt. The system claims a recognition if it finds a database object whose affine-transformed features show a sufficiently small rms error when compared to the new-image features.

Machine-vision rescue

Examples of packages must be programmed into a training system to teach ViPR to recognize and associate them with the appropriate UPC codes (see Fig. 2). The grocery chain finds that it has 300-500 items that shoppers are likely to put on the bottom of its carts. It only needs to train the system to recognize these items-a fraction of the tens of thousands of items that the stores stock.

FIGURE 2. The visual scanner system integrates into the POS as a node on an Ethernet LAN that provides connection between the POS host and peripherals such as barcode scanners and credit card swipers. The scanner system includes custom lighting and a USB 2.0 camera unit built into the bottom of the checkout counter and COU computer. ViPR software running on the COU performs image-recognition tasks as calls from a C++ program.

Since training produces a database entry, it is not necessary for each store to train its system on each BOB-likely item it stocks. Once the system has been trained to recognize an item, the information can be distributed electronically to all the LaneHawk systems in all the stores that carry the item.

The LaneHawk provides the POS with a UPC matching each BOB item it recognizes. Then the POS can query the price database for the item price and create a transaction log entry for the item. The cashier must verify the transaction log entry and ring up the sale.

When the LaneHawk recognizes a BOB item, it alerts the cashier on the POS human-machine interface and overlays a “Process BOB Items” soft key onto the display’s “Total” key, preventing the cashier from completing the checkout without processing the items (see Fig. 3). The cashier can still ring up items crossing the counter using the scanner and other peripherals but cannot complete the checkout without processing the BOB items.

The cashier can modify the transaction, reject it, or accept it. By accepting the transaction, the cashier posts the transaction log entry. Once all of the BOB items have been processed, that screen closes and the overlayed button disappears from the display revealing the “Total” soft key. The cashier can then press that key to log the transaction (see Fig. 4).

FIGURE 4. The BOB Items screen lists all of the items LaneHawk recognized as products needing to be checked out. The cashier then can verify and add them to the transaction record. After processing the BOB items, the cashier returns to the main POS screen to complete the checkout process.

The LaneHawk software runs on a vision-acquisition and processing hardware package produced by business-equipment supplier Ricoh. The package consists of a lighting-and-camera unit (LCU) that mounts in the checkout-counter base to observe the bottom rack as a cart rolls through the lane and a checkout unit (COU) where images and data are processed. The COU checks to see if there are any packages in the rack and, if so, identifies them by UPC code. It then sends that information via an Ethernet cable to the POS checkout system. The POS alerts the clerk and prevents final checkout until the items are verified and added to the sale total.

Ricoh used a customized 640 × 480 Lumenera camera with a wide-angle lens, which connects directly via USB 2.0 to the processor-an Intel Pentium 4 running at a clock speed greater than 2 GHz under Linux. The LCU also contains a proprietary white-light LED array and a photodetector to ensure adequate illumination under the cart.

The COU houses the system host computer running the object-recognition software and the applications that connect it to the POS Ethernet. Any software updates or changes to the database of BOB-likely items can be downloaded through the POS system interface to the store intranet.

Company Info

Evolution Robotics
Pasadena, CA, USA
www.evolution.com

Lumenera
Ottawa, ON, Canada
www.lumenera.com

Ricoh
West Caldwell, NJ, USA
www.ricoh-usa.com