Using networked smart cameras, a vision system deployed at a cigarette manufacturer ensures that boxes of tobacco can be tracked through the supply chain.
Faisal Mirza
In the packaging industry, corrugated cardboard boxes filled with tobacco products must be assigned a unique number prior to shipment so that they can be accurately tracked through the supply chain.
In an effort to streamline the process, one company with a tobacco packaging facility in Pakistan is now employing two vision-based solutions developed by Sunshine Corporation (Karachi, Pakistan; www.sunshine.com.pk). The systems can identify a variety of different brands of tobacco products contained in each of the boxes, after which a unique serial number is automatically printed on them together with a time stamp, their weight and environmental variables such as temperature and relative humidity.
The inspection process presented a challenge because there were a large number of different boxes of products all conveyed on the same line with very little difference between them in terms of shape, size and color. What is more, there were no existing barcodes printed on the cartons that could be used for identification purposes. In addition, it was imperative to ensure that the two sets of vision inspection and printing systems interfaced with one another to ensure that a unique number would be printed onto each box.
With the support of the Planet NI program from National Instruments (Austin, TX, USA; www.ni.com), Sunshine Corporation built a networked solution to deal with the high volume production demands of the customer. The system distinguishes the boxes according to the unique patterns, designs, or logos that are printed on the outside of the boxes, can accommodate up to 30 boxes each with a unique logo on the same line if necessary, and can operate continuously throughout the year, 24 hours a day (Figure 1).
On the line
In the packaging plant, the boxes containing the different varieties of tobacco products are individually transported from an in-feed conveyor onto a variable speed conveyor equipped with four shear beam load cells from Axis Weighing (Gifu Ken Fuwa Gun, Japan; www.axisweighing.com) where the total weight of the container is measured and streamed to a Koyo Click PLC supplied by Automation Direct (Cumming, GA, USA; www.automationdirect.com).
To ensure the accuracy of the weighing process, it was important to guarantee that no more than one carton of tobacco products entered the weighing station at a time. To do so, a model QS18VP6LVQ5 retro-reflective sensor from Banner Engineering (Minneapolis, MN, USA; www.bannerengineering.com) was mounted at the in-feed of the load cell conveyor and interfaced to the PLC. Should a carton trigger the in-feed sensor while another carton is already present on the load cell conveyor, the motion of the in-feed conveyor is halted, ensuring that only the weight of a single carton is measured (Figure 2).
Once in the field of view of the vision inspection station mounted above the conveyor, a second model QS18VP6LVQ5 retro-reflective sensor instructs the PLC to trigger a National Instruments 1774C 1280 x 960 (SXGA), 17 fps color camera with an onboard 1.6 GHz processor. The camera captures an image of the artwork on the top of the carton from which the camera's embedded software identifies the box and associates a product ID number to it that is specific to the brand.
The product ID number is then transferred to a 10in industrial touch panel PC from Winmate (New Taipei City, Taiwan, ROC; www.winmate.com.tw). Here, the number is combined with weight data from the load cell conveyor and the temperature and relative humidity data acquired by the PLC from a HX94C sensor/transducer from Omega (Stamford, CT, USA; www.omega.com). Together, the sets of data are used to create a unique code that is printed on the box by an inkjet printer as it moves down the line.
Vision inspection
The NI 1774C smart cameras were chosen for the application because they are IP67 rated and can withstand harsh industrial environments. Furthermore, their onboard image processing capability relieves the PC from any image processing tasks, which is relegated to handle network communications and printing of the code.
The cameras can be programmed to identify numerous different types of boxes by processing the captured images using image processing tools in National Instruments' interactive Vision Builder for Automated Inspection software environment. Prior to inspecting any new product, a 1774C camera must be trained to identify the unique logos found on the outside of the box. To do so, a color pattern matching tool is used to create a template of each of the color features printed onto the cartons, after which the template is assigned a unique identification number together with the name of a particular brand of product.
Once an image of the current box is captured by the camera on the packaging line, a tool within the Vision Builder first locates the edges of the carton in the image. The image is then cropped to provide a region of interest in the image from which the location of the identifying brand feature can be determined. Having cropped the image, the software running on the camera uses a color pattern matching tool to determine the color and spatial information in the patterns in the color image.
A search tool then searches through the templates taught for each new product that are stored in the memory of the camera to find a match in a rotation invariant fashion - an important feature, since each carton could be rotated by 180° by the case packers on the production floor. Having found the most likely fit, the software then scores a match (on a scale of 0 to 1000) for the current pattern against the original template for each product. If the score is high enough (above 750), the camera transfers the image together with a number identifying the product to the PC. If the score is too low (below 750), or if two or more templates match the same product (which rarely happens), the carton is flagged as having failed the inspection process and is tagged for rejection further down the conveyor line.
The software running on the PC then checks to determine the last number that the system had allocated and printed onto cartons containing similar tobacco products. It then increments the older number, and stores it on the PC together with the weight, temperature and humidity data acquired by the PLC and the time and date at which the box of products was inspected.
Because there are two vision-based inspection and printing stations installed at different locations in the facility which are both inspecting cardboard cartons of tobacco products, at any instant, it could be possible that two boxes containing the same brand are being inspected simultaneously. Hence it was important to ensure that unique identification numbers were printed on each of the boxes so that no two boxes with the same brand of product inside them ever have the same count printed on them. To do so, the PCs controlling each system communicate with one another over an Ethernet connection, and through the use of shared software variables and queues, the PCs can correlate the two sets of data in real time, incrementing any duplicates to ensure that each number assigned to a box is unique.
The software user interface for the system was built using National Instruments LabVIEW software which works with the company's Vision Builder software. LabVIEW was used to create an HMI interface that runs on a touch panel PC display by Winmate, enabling an operator to view the results of the image acquisition and analysis process, together with the number and type of cartons that were inspected in a given period of time. In addition, the software displays the name of each carton and the pass and failure rates associates with each type of brand (Figure 3).
The maximum length of the largest box that runs on the conveyor is 560mm. Hence, area lights with sufficient intensity and large covering area were required to produce homogenous illumination. Since the area of interest in the images contain multi-color patterns, it was also necessary to use white light illumination to produce the best results. For this reason, two 580 mm model LEDWLA580XD6-XQ linear array area lights from Banner Engineering mounted below the camera were employed to directly illuminate the target.
Rejection or acceptance
Having passed the inspection process, the box is transported towards a C-1000 inkjet printer with an internal diffuse mode trigger sensor from Domino Printing (Cambridge, UK; www.domino-printing.com). While the box is in transit, the PC transfers the unique identifying number, weight, temperature, humidity and time and date to the printer over an Ethernet interface. Once the presence of the box is detected by the sensor, the inkjet printer prints the data along with a 2D QR code onto the outside of the box (Figure 4). No number, however, is printed on those boxes that cannot be accurately identified by the vision system.
A two-liter ink bottle inside the printer is sufficient for months of printing on the cartons. However, to avoid the possibility of very low quality printing (or no printing at all) due to low ink levels, the box moves under an iVu pattern match camera from Banner Engineering whilst being illuminated by a LEDWA80X80M 80 x 80 mm area light. The pattern match camera runs a software tool that searches for a given quantity of dark pixels (ink) in the image that contrast with the light background of the carton. Images with a high contrast ratio indicate the presence of the printed data, whilst those with a low contrast ratio indicate that the printing process has been unsuccessful.
Based on the measurement of the contrast ratio, a pass or fail signal is generated and returned to the PLC. To reject the carton, a photoelectric sensor is triggered further down the line which instructs the PLC to reject any box that has either failed to be printed, or was not able to be identified by the 1774 camera. It does so by actuating a linear cylinder which pushes the box from the conveyor into a rejection station. Rejected cartons can then be manually inspected to determine the specific nature of the failure and the products inside them repackaged if necessary.
Since they were installed at the packaging plant, the two systems have been running continuously throughout the year despite the fact that they are exposed to large variations in both ambient lighting and temperature during the summer and winter months. The systems -- which can each inspect and print around 10,000 corrugated box cartons of tobacco products per day -- have effectively replaced earlier manual methods which were employed to serialize such products, reducing inefficiencies and errors that had previously led to costly recalls or rework.
Faisal Mirza,
Proprietor, Sunshine Corporation, Karachi, Pakistan
Companies mentioned
Automation Direct
Cumming, GA, USA
www.automationdirect.com
Axis Weighing System Co.
Gifu Ken Fuwa Gun, Japan
www.axisweighing.com
Banner Engineering
Minneapolis, MN, USA
www.bannerengineering.com
Domino Printing
Cambridge, UK
www.domino-printing.com
National Instruments
Austin, TX, USA
www.ni.com
Omega
Stamford, CT, USA
www.omega.com
Sunshine Corporation
Karachi, Pakistan
www.sunshine.com.pk
Winmate
New Taipei City, Taiwan
www.winmate.com.tw