Vision system simplifies robotic fruit picking

Robotic apple picker relies on a camera inside the gripper and off-the-shelf components and software.

By Winn Hardin, Contributing Editor

In the last few years, Belgian farmers have experienced problems with finding reliable seasonal workers for harvesting choice fruit, including apples. In response to this problem, the Automation Centre for Research and Education (ACRO) Institute developed a robotic apple harvester. ACRO is a spin-off company and technical consultant to the Catholic University College of Limburg, Belgium, and also provides PLC and machine-vision technical training and research to the European community. ACRO is partially funded by the Belgian government.

FIGURE 1. Horizontal 2-D stabilization of the APFM consists of two hydraulic feet and one turnover cylinder, configured as a three-point suspension. Controlled by two level sensors, the system ensures stable positioning during picking. The white funnel-formed gripper unit is at the end of the robot arm.

The European standard height of fruit trees on professional plantations must be between two and three meters. The ACRO automated fruit-picking machine (AFPM) harvester uses a unique vacuum-gripper design to pick the fruit and ease coordination between the vision system and robot controller. Mounted behind a common agriculture tractor, the AFPM platform supports a Panasonic industrial robot to “pick” the fruit (see Fig. 1).

The AFPM apple harvester also consists of a tractor-driven generator for power supply, a horizontal stabilization unit, a seventh external vertical axis to enlarge the operation range, a SICK safety scanning device, a Siemens central control unit, and touch panel PC with human-machine interface (HMI). MVTec Software’s Halcon image processing software provides the robot guidance coordinates, while a canopy and curtain, which can be folded up during transportation, reduce the affects of ambient light (see Fig. 2). The AFPM needs one driver on the tractor while it effectively handles the workload of six workers.

CAMERA IN GRIPPER

One of the most challenging problems was the design of the fruit gripper. An entirely new end-of-arm tool (EOT) had to be developed to pick the apples with extreme precision and care. Not only did this device have to assume the unique form of each apple, but the force used to “pick” the fruit could not damage the fruit. ACRO’s engineers decided to create a vacuum-based gripper that could also ease the vision system’s task of guiding the Panasonic VR006L six-axis robot by concentrically locating a digital camera inside the cone of the vacuum suction cup. With these design goals in mind, the final gripper design is a combination of a white flexible silicone cone surrounding an IDS UI-2230RE-C uEye USB 2.0 color camera. The cone is fed by a reversible vacuum/blower (see Fig. 3).

FIGURE 3. Vacuum gripper unit of the APFM has a USB 2.0 camera inside.

By placing the camera inside the gripper, the position of the camera is fully controllable because it is fixed to the arm of six-axis robot. The camera can point its optical axis at the apple, reducing image distortion and eliminating repetitive calibration steps during apple picking. A final advantage is that the camera is protected against collisions or bad weather conditions, as well as against direct sunlight.

FRUIT PICKING

Once the apple harvester is moved in front of the tree and its canopy opened around the AFPM platform (see Fig. 4), the harvester has to be actively stabilized. For this purpose, the machine is equipped with different hydraulic foot and elevator systems for front-behind and left-right movements. These stabilizers are controlled by electronics bundled with the hydraulic system and automatically controlled through the use of water level sensors.

FIGURE 4. The APFM picks apples in the orchard. The canopy covers an apple tree against intense sunlight.

After the rig is adjusted, the camera scans the tree from 40 preprogrammed positions. In this manner, the device acquires a grid of images of the front of the tree with the grid broken into horizontal and vertical “blocks” created by panning the camera across in increments of 10°, and then panning it down by 10° and beginning the next grid row. Thus, each tree is divided into 40 sectors or images.

The programmed location and orientation of the robotic arm is stored with each image. Location data is stored in the PC’s RAM. For each sector, all ripe apples are identified by the image processing software, listed, and picked one by one in a looped task.

TARGETING FRESH FRUIT

To improve the efficacy of the image processing algorithms in finding and locating the apples on the tree, the platform was designed to control the lighting conditions to the greatest extent possible, using a canopy to cover the entire tree and the AFPM platform to reduce the effects of changing ambient lighting conditions and provide a uniform background (blue) to ease locating the (red and green) apples.

Image processing is conducted on an industrial PC with 2 GHz Pentium IV microprocessor and 1 Gbyte RAM running Windows XP. For image-processing software, the AFPM designers elected the standard machine vision library Halcon 7.1 from MVTec because of its accuracy and reliability, according to Eric Claesen, lead designer for the project at ACRO. The system can pick 85% of all apples of a tree, similar to the achievements of manual picking operations.

During system calibration, the first step is to train the system on the color of the apple tree’s leaves and the blue background canopy. A series of color thresholding steps are used to filter out these unwanted features. First the blue is filtered out of each of the 40 images that represent one side of the tree, and then the leaves are filtered out. The green and the red parts of the apples’ surfaces are located via color thresholding. After noise reduction, using Halcon’s opening_circle and select_shape convolutions and filling holes in the images, the apples’ positions are clearly emerged (see Fig. 5). After separating the individual apples from clusters (when present) by watershed-filtering, each apple is selected and transformed into a circle.

ITERATIVE GUIDANCE

To pick one apple, the robot has to determine the distance between the camera and the apple and the path to get there. The camera measures this distance by triangulation. The measuring is done in several steps. The camera first acquires an image, and then the camera is turned so that the apple is situated in the center (see Fig. 6). The camera then acquires a second image, and finally the diameter is calculated by processing these two images.

FIGURE 6. Distance between camera and apple is measured by triangulation, and robot guidance is performed on iterative “Stop and Look” method based on extracting position data from periodic, static images rather than continuous monitoring of real-time video of each apple common to visual servoing.

At this point, the vision system determines if the apple falls within the acceptable size range. If so, a signal is sent across a Profibus connection to the robot controller, and the robot arm is allowed to continue toward the apple.

While approaching the apple, several images are processed to calculate by triangulation the remaining distance to the apple while air is blowing through the gripper at about 300 m³/min to free the fruit from leaves that might conceal it. With each image acquired, the system calculates the remaining distance using a proprietary formula.

Since the apple remains centered in the image, the correlation of a given apple in subsequent images is trivial. When the selected apple is approached, air blows through the picking device to clear the apple from leaves. Once the apple is within a well-defined range of the gripper, the vacuum device is activated. If a vacuum is detected, the apple is rotated and tilted softly, picked and then put beside.

Features, advantages, benefits

Says ACRO’s Eric Claesen, “The goal, at the beginning-three years ago-of the project, was to research if it was possible to pick an apple with the technology of today. Now at the end of the project we can say that the project succeeded. We can detect and pick 80% of the apples in a tree correctly. At the moment the picking machine picks an apple every 8 s but this will be reduced to 6 or 5 s after the pick season in 2007.

“The designers believe that this time span can be lowered to about 5 s, chiefly by reducing the communication bottleneck between the vision system and the central controller unit. The designers are also planning to better the automation of the navigation through the orchard and to make the device suitable for picking other fruit, such as pears.”