Norfolk Southern’s Machine Vision System Inspects Moving Rail Cars

To spot defects on moving trains, Norfolk Southern Railway (Atlanta, GA, USA) is building train inspection portals on tracks in the United States.

The Georgia Tech Research Institute (Atlanta, GA, USA) is collaborating with the railroad on the inspection project.

The portals combine high-resolution cameras, stadium lights, and AI algorithms to capture and then analyze images of trains moving at up to 60 mph. All components are installed trackside inside tunnel-type steel structures.

Two inspection portals were deployed in 2023 on adjacent tracks in Leetonia, Ohio—not far from where a train derailed in February 2023, spilling hazardous chemicals. Trains pass through the new inspection portals about once per hour.

Norfolk Southern plans to deploy a dozen automated portals by the end of 2024 to augment, not replace, human inspectors.  

 “Human inspections consist of experts walking the length of the train at various points along the rail network, monitoring for defects. As you can imagine, this process can be time-consuming and subject to uncontrollable conditions—like weather,” says Mabby Amouie, chief data scientist at Norfolk Southern.

In addition to weather resistance, the automated system provides the railway with information about potential problems that may be difficult for humans to uncover. “One advantage of the automated inspection portal is that the trains can be inspected in motion, under forces that can expose faults that are not obvious when the train is sitting still,” says Colin Usher, senior research scientist, Georgia Tech Research Institute (GTRI).

There are a variety of defects that Norfolk Southern wants to detect. “As big as trains are, surprisingly, it comes down to relatively small bolts and cotter pins located in various locations to keep everything working properly. In addition, safety equipment, such as ladders, is going to be inspected for damage. Finally, items such as open hatches or open doors are also going to be inspected,” explains Usher.

Machine Vision and AI

Engineers at GTRI designed the hardware setup for the new inspection portals, and Norfolk Southern’s employees created the AI-enabled inspection algorithms.

The system includes 38 cameras, which are installed strategically at various angles, including some at track level pointing up, to capture specific points on each rail car.

The cameras include both area and line scan models, and the combination produces approximately 1,000 images per rail car, producing a 360° view of each rail car.

The cameras and lenses are installed inside protective enclosures and mounted on arms that are attached to the sides of the tunnel structure.

Stadium lights are mounted inside the tunnel enclosure along the sides and top.  “To capture clear images of trains travelling at up to 60 mph requires extremely fast shutter speeds, which dictates that we provide lots of light to illuminate the trains,” Usher says.

Sensors that detect the presence of an oncoming train and its speed, trigger the entire process. Based on the speed, software developed at GTRI calculates when each camera should take images and lights should turn on. Cameras are synched to a microsecond. “Only images of critical components are taken and the other areas of the train that are inconsequential to identifying defects are not captured. That optimizes the image capture and saves space in the computer system,” Usher says.

To further save space, image data is compressed.

AI-enabled algorithms analyze image data on-site to determine if defects are present. Within minutes, results are then transmitted to Norfolk Southern’s network operations center, where subject-matter experts review the data. Critical defects are flagged by the AI software for an immediate response, such as scheduling the train for service at the next train yard or stopping immediately if the problem is severe.

Norfolk Southern’s inspection algorithms “have very high detection rates, while also having very low false alarms,” adds Amouie.

Machine Vision Design Challenges

Usher says the team overcame numerous obstacles in the design of the system.

“We encountered challenges with data throughput, processing times, and storage.  Each train results in an average of 300 gigabytes of data and this is after significant image compression,” Usher says.

To solve the problem, Usher continues, “we use enterprise systems similar to those in data centers to manage the data and processing requirements.” Multiple enterprise-level computers are located inside the power supply cabinet at each inspection portal.

Another challenge was protecting the components—such as cameras, lenses, and lights—from outdoor weather as well as vibration caused by the passing trains.

For example, the enclosures are made of steel, which not only protects the components from inclement weather but also blocks direct sunlight from the imaging sensors “so as not to cause bloom on the images,” Usher says.

Similarly, stadium-type lighting is designed to withstand outdoor environments.

When asked what is unique about the train inspection system, compared with others in use in the railroad industry, Usher says, “This system adds numerous additional camera angles and views of the train cars and components that we are unaware of in other systems. We also use higher resolution and high-speed cameras and a combination of area and line scan cameras to generate comprehensive views of components for looking at specific types of defects.”