3D Imaging: 3D vision and neural networks team up to sort tuna fish

Commercial tuna fishermen use large shipping vessels that are capable of staying at sea for long periods of time, storing their catch on-board in large freezer compartments. Once the vessels are docked in port, the tuna must be offloaded, transferred to processing plants where they are sorted by type, size and quality before being shipped to the customer.

Marexi has designed and developed a 3D imaging system, known as TUNASCAN that can perform the task of classifying and sorting tuna fish.

"In the past," says Joaquín Gracia Salvador, President of Marexi (Vigo, Pontevedra, Spain; www.marexi.com), "large processing facilities capable of processing up to 200 Tonnes/day required more than 20 people to classify the tuna." Operators would have to classify various types of tuna such as Big Eye, Yellow Fin and Skip Jack by type, weight and quality condition. Needless to say, this task was time consuming, labor intensive and subject to human error. Because of this, Marexi has designed and developed a 3D imaging system, known as TUNASCAN, that can perform this classification and sorting task automatically.

"While sorting these fish manually is only performed 40% accurately," says Salvador, "with the TUNASCAN system, the accuracy is improved by over eight times." In this way, the customer is supplied with the correct product (quality and species) that can be priced accordingly.

To perform this classification and sorting task, frozen tuna fish are first loaded into a large holding tank where they must be separated prior to inspection. This is accomplished by loading the fish from the tank onto a 3.5m diameter centrifugal plate that rotates at 3m/s. As the plate rotates, centripetal force forces individual tuna to the outside of the plate. To singulate the fish onto a conveyor, a pneumatic actuator opens a side door on the plate housing, allowing individual tuna to pass through at fixed intervals. The tuna fish are then transported onto a conveyor belt into the TUNASCAN's machine vision system.

Using a graphical user interface, an operator can choose the species, size and weight and determine how the fish should be sorted into the multiple bins located along the conveyor.

"To characterize which type of tuna is present," says Salvador, "over 80 different types of measurements must be made." These include weight, the shape and size of the fish as well as more complex measurements such as the distance between the head and the dorsal fin.

Making these measurements required using a structured-light based imaging system to create a 3D rendition of each fish. As each fish enters the vision system, it is illuminated on both sides by two 3mm width line lasers mounted off-axis across the approximately 70mm conveyor. Although the frequency of the laser light remains proprietary, it was chosen to optimize the amount of reflected light from often ice-covered tuna.

(Top) As each fish enters the vision system, it is illuminated on both sides by two 3mm width line lasers mounted off-axis across the approximately 70mm conveyor. (Bottom) 3DExpress is then used to generate a metrically-calibrated Cloud-of-Points (COP) from the laser triangulation system.

To capture this reflected light, two custom built cameras from Photonfocus (Lachen, Switzerland; www.photonfocus.com) mounted at opposing off-axis angles were used. Line-profile data from each of these cameras was then transferred over the cameras' Ethernet interface to a host PC. There, 3DExpress from Aqsense (Girona, Spain; www.aqsense.com ) is used to generate a metrically-calibrated Cloud-of-Points (COP) from the laser triangulation system. Tools within 3DExpress are then use to make feature measurements of the image to determine its weight, shape and size and distances between known features.

"Once these measurements have been made," says Salvador, "they can be used to classify which fish type is present." To accomplish this, the 80 extracted characteristics are classified using an artificial neural network (ANN), custom designed by Marexi to run on an NVIDIA co-processor in the PC. "Using this processor, the system can classify each fish in approximately 200ms," says Salvador.

Of course, such ANN-based systems need training before they can be used effectively. To do so, the system was first trained with multiple known good samples of Big Eye, Yellow Fin and Skip Jack tuna. Once trained with these examples, the system extracts variable features from unknown samples and the ANN is used to weight these features and classify the tuna with an approximate 95% accuracy.

Once classified, the fish must be sorted into relevant bins by type and weight. Should a tuna be deemed unacceptable due to effects such as crushing or other deformities, it is transported to the end of the conveyor. As acceptable tuna moves from the vision system along the conveyor, the CPU actuates a programmable logic controller (PLC) that is interfaced to deflection plates on each side of the conveyor. Depending on the type and weight of each fish, these deflection plates guide the tuna to the appropriate bin. Here, an acoustic sensor, also interfaced to the PLC is used to detect that the fish has been sorted. This data is then fed to the PC over an Ethernet interface using an Industrial Protocol IP.

To configure the TUNASCAN system, Marexi has designed a touch panel graphical user interface (GUI) written in Visual C++ that an operator can use to choose the species, size and weight. This GUI can also be used to determine how the fish should be sorted into the multiple bins located along the conveyor. TUNASCAN systems are now marketed exclusively by Hermasa (Vigo, Spain; www.hermasa.com) and are in operation in numerous locations throughout Europe, Asia, South America and the United States.