Multisensor imaging aids non-ferrous scrap metal sorting

Andrew Wilson, European Editor

Vehicles and electrical and electronic equipment are often discarded as waste since it may be impractical to recycle this material in a cost-effective manner. In the past, such recycling was performed by manual sorting-a procedure which was labor intensive, time consuming and inaccurate. Using automated systems, 40-50% of the steel that comes from recycled material and 30% of other metals can be sorted by magnetic drums or induction or Eddy current separators, respectively. One of the greatest challenges in developing more sophisticated systems is sorting various non-ferrous metals such as aluminium, copper, silver, brass, lead, stainless steel, silver and gold.

Traditionally, the recycling industry has been slow to adopt automated methods of sorting such metallic scraps. However, with the advent of hyperspectal cameras and robotic-based picking and sorting methods, such procedures can now be performed automatically, relieving workers of laborious tasks while at the same time improving the efficiency of sorting metallic alloys. Since the automation process is more effective, scrap providers can produce more recycled material thus increase their profitability.

To achieve these goals, the GeMMe unit of the University of Liè;ge (Liè;ge, Belgium;www.gemme.ulg.ac.be), a group initially dedicated to mineral characterization and processing, has developed a robotics based system that combines multiple sensors, cameras, programmable logic controllers (PLCs) and robotics systems (Figure 1). "Sorting of scrap metal pieces is a challenging problem due to the flat reflection response of metals," says Pierre Barnabé, a Research Engineer of the group, "indeed, the reflectance curves of metals typically present monotonous characteristics, as opposed to plastics that may have more singular spectral shapes."

Hyperspectral imaging

To perform the task of sorting non-ferrous metals first requires the material to be located and then classified based on its spectral signature. These spectral signatures can then be used to identify the material. Aluminum, copper, zinc, stainless steel and brass metal all exhibit different spectral response characteristics in the visible and near infrared (VNIR) spectrum (Figure 2). As can be seen, brass and stainless steel are the most difficult classes to distinguish since their spectral reflectance characteristics are similar. Because of this, hyperspectral imaging systems must be used to classify each individual material.

Figure 2: Aluminum, copper, zinc, stainless steel and brass metal all exhibit different spectral response characteristics in the very near infra-red (VNIR) spectrum. Brass and stainless steel are the most difficult classes to distinguish since their spectral reflectance characteristics are similar.

Before classification, however, non-ferrous metals are tracked as they move through the sorting system. In the system developed at the University of Liè;ge, this is performed using a 3D Ranger camera from SICK (Waldkirch, Germany;www.sick.com) mounted above a conveyor belt. Using this camera, images of particles are segmented and the size and position of each metallic part is measured. Data from the camera is then transmitted over a Gigabit Ethernet interface to a host PC.

In the design of the system, a programmable logic contorller (PLC) from Beckhoff (Verl, Germany;www.beckhoff.com) tracks the conveyor progress and is used to trigger air blowers at the end of the conveyor. The incoming signal from the encoder is interfaced to several frequency dividers from Motrona (Gottmadingen, Germany; www.motrona.com) and then to the cameras, frame grabbers and the sorting devices' PLCs. Software then sends positional coordinates to the robots' PLCs through a TCP interface such that there is no PLC between the encoder and sorting devices/sensors, only frequency dividers.

"To classify non-ferrous metals, a significant portion of the reflectance spectrum must be measured to maximize the system's discriminative potential," says Barnabé. For this reason, two hyperspectral cameras are used to acquire images. These consist of a MV1-D1312IE-Camera Link camera from Photonfocus (Lachen, Switzerland;www.photonfocus.com) with a spectral response in the visible and near infrared (VNIR) region. This camera is mounted with a Xenoplan 1.9/35mm broadband lens from Schneider Optics (Hauppauge, NY, USA; www.schneideroptics.com) coated for imaging between 400-1000nm. This camera is interfaced to a host PC using a Camera Link frame grabber from Imperx (Boca Raton, FL, USA; www.imperx.com). The second camera, an SWIR camera from Specim (Oulu, Finland; www.specim.fi), is used to image the non-ferrous metals in the 1000-2500nm spectrum and is interfaced to the host PC using a Camera Link frame grabber from National Instruments (Austin, TX, USA; www.ni.com).

While both the cameras feature area array sensors, they are instead used in line-scan (or push-broom) mode. Coupled with a ImSpectorV10E spectrograph from Specim, the VNIR camera images the metallic particles such that the detector registers the spatial position from a line and the spectral information in each spatial position. To maximize the amount of light for each spatial pixel, spectral binning is performed with the Photonfocus camera, stacking eight adjacent pixels. This has the benefit of reducing the spectral resolution of the sensor to the same magnitude as the intrinsic resolution of the spectrograph (that is approximately 5nm).

To illuminate the metallic parts as they move along the conveyor at speed, a high-level of illumination was required. For this reason, an extruded elliptical reflector was used. This consists of an array of Halogen lamps such that light from the striking reflector are redirected to converge and produce a concentrated beam in a linear fashion (Figure 3).

Merging images

After line-scan images are captured by the VNIR and SWIR cameras, they must be merged. The spatial resolution for the VNIR camera is 1312 pixels and 320 pixels for the MCT-based SWIR cameras across the line-scan image. To merge these images, the value of surrounding pixel's geometric mean from both the VNIR and SWIR camera is combined to produce a final image.

After images are captured by the system, classification algorithms are used to analyze the reflectance from the metal, allowing the system to classify them correctly. To do so, the perClass software from PR Sys Design (Delft, The Netherlands;www.perclass.com) with its Matlab-based perClass Toolbox was used to analyze different trained classifiers and thus develop an optimal solution. These included Quadratric and Mixture Gaussian classifiers, k-nearest neighbor classifiers, Fisher discriminant analysis, and a Random Forest method.

Before classification, however, linear discriminant analysis (LDA) was applied to the data gathered from the hyperspectral system. LDA can be used to find a linear combination of features that characterizes or separates two or more classes of objects. Based on the features extracted from the data with LDA, the Random Forest algorithm provided the most promising results for metallic alloy classification (Figure 4). Such Random Forest methods are supervised training classifiers that operate by first building a multitude of decision trees as the system is trained. The classifier then determines the class that appears most often.

Figure 4: Based on the features extracted from the data with LDA, the Random Forest algorithm provided the most promising results for metallic alloy classification.

Robotic sorting

As the metallic alloys are deposited on the conveyor, they are identified by the 3D Ranger camera and then classified by the hyperspectral system. Using a controller from ABB (Zürich, Switzerland;www.new.abb.com) synchronized to the conveyor belt's encoder, these particles are tracked as they move through the system. Software routines written in RAPID, a high-level programming language used to control ABB industrial robots, are then used to determine the specific position of a each alloy after it is identified by the 3D camera. "This allows the ConveyorTracking module of ABB software to provide instructions to the ABB picking robots without having to take into account the progression of the conveyor," says Barnabé.

Object locations are converted from the 3D camera image into a moving referential to the ABB robot using a periodic pulse from the conveyor's encoder. With ABB's ConveyorTracking module, movement instructions are then referenced to the robot.

In the system, two IRB 340 ABB robots are used for sorting. While one uses a 2-fingered gripper, another uses a bucket to slide objects over the conveyor while avoiding other objects. Both are interfaced to the system using a TCP connection between the acquisition computer and each robot controller.

At the end of the conveyor, pneumatic ejection devices are also used to sort material and metallic alloys. To identify optimal picking points for the robot, a custom algorithm is used while constrained Delaunay and A* algorithms are used for path finding.

In this case, software routines in the ABB Controller and the PLC controlling the air blowers store the instructions received from the processing computer, and perform the actions when object is in range. A video of the systems in operation can be found at:http://bit.ly/VSD-SCRAP. While at present, the system uses VNIR and SWIR cameras to capture image data, Barnabé and his colleagues are currently integrating other types of sensors for more advanced classification.

While the current system uses 3D and spectral cameras, Barnabé and his colleagues are integrating X-Ray transmission (XRT) sensors and Laser Induced Breakdown Spectroscopy (LIBS) sensors to assess the best sensor combination for different products and applications. While the XRT will be used to find, for instance, steel bolts inside aluminium scrap, the LIBS sensor will detect the composition of different aluminium or stainless-steel alloys.

Organizations and companies mentioned

ABB
Zürich, Switzerland
www.new.abb.com

Beckhoff
Verl, Germany
www.beckhoff.com

Imperx
Boca Raton, FL, USA
www.imperx.com

Motrona
Gottmadingen, Germany
www.motrona.com

National Instruments
Austin, TX, USA
www.ni.com

Photonfocus
Lachen, Switzerland
www.photonfocus.com

PR Sys Design
Delft, The Netherlands
www.perclass.com

Schneider Optics
Hauppauge, NY, USA
www.schneideroptics.com

Specim
Oulu, Finland
www.specim.fi

University of Liège
Liège, Belgium
www.gemme.ulg.ac.be