Drones, High Resolution Cameras Boost Inspection Safety and Quality

HHLA (Hamburg, Germany), through its subsidiary company, HHLA Sky, recently implemented a process utilizing drones for inspecting the giant gantry cranes used for loading and unloading cargo containers in the Port of Hamburg, Germany. 

HHLA, a European intermodal logistics company, operates three cargo terminals at the Port of Hamburg as well as terminals in other European ports.   

The third largest port in Europe, the Port of Hamburg handled about 82.3 million tonnes (a tonne is about 2,200 pounds) of cargo in 2022, and some 8.3 million TEUs— a unit of measure that approximately describes a single 20 ft by 8 ft cargo shipping container. 

In order to handle all those containers, HHLA utilizes some of the world’s largest container gantry cranes—more than 50—to unload cargo containers from ships. Each crane weighs more than 2,000 tonnes and has a lifting arm some 80 meters long. The crane, after lifting a container from a cargo ship, lowers it onto an autonomous guided vehicle. The AGV then autonomously drives the container to a depot. From there, rail mounted cranes (RMGs) pick up the container and place it on a train or truck that in turn transports the container to its next destination.  

This process runs 24 hours a day, seven days a week, 365 days a year. That means each crane requires significant maintenance, including two required safety inspections a year. Obviously, safety is of the utmost importance in a commercial port, where people work with huge pieces of moving machinery and massive amounts of cargo every day.  

Up until recently, HHLA had human industrial climbers conduct these inspections. Not only is the job of climbing ladders and scaffolds to manually inspect the cranes inherently a slow, methodical and dangerous undertaking, but the entire workflow at the port slows significantly, creating a bottleneck in operations, because each crane is taken out of service during the inspection process.  

HHLA began to seek safer, quicker, easier alternatives. This led to research into using drones, machine vision cameras and software to automate inspection of the cranes. This ultimately resulted in the formation of subsidiary company, HHLA Sky, which developed and now markets the drone-based automatic inspection system. 

Related: Vision Solution for Drone Collision Avoidance  

HHLA Sky CEO Matthias Gronstedtnotes that drones are effective automated inspection platforms, as they can fly precisely, repeatably and often in dangerous or hard to reach places, in harsh weather conditions. They can eliminate the need for ladders, scaffolding and platforms while boosting efficiency and ease of use.  

How it Works 

At the Port of Hamburg specifically, Gronstedt says the HHLA team uses commercial drones that are equipped with Sony A7RIV and A7R V cameras, mounted to the drones using a mounting bracket, known as a gimbal, made by Gremsy (Ho Chi Minh City, Vietnam). Each camera is equipped with 61 MPixel full frame sensors and Sony high-precision E-mount lenses. Using these cameras, the HHLA team can fly a drone at least 15 m away from a crane and still capture detailed image data. 

"The pictures are so good that we can detect even tiny scratches on the cranes,” Gronstedt says. “We use the native 61 MPixel resolution to let us digitally zoom in and still have a sharp image, and we continue to develop AI models to recognize defects and automatically prepare reports.” 

HHLA Sky has also developed and implemented mobile robot management software called the Integrated Control Center, an IoT system which can control and track more than 100 drones and autonomous mobile robots at the same time, as well as AI models to help identify potential safety issues on the cranes.  

 At the port in Hamburg, each drone is programmed to fly a pattern around one or more cranes and capture images at multiple neuralgic points. These images are transmitted wirelessly to the Integrated Control Center, located separately from the drone base station, via an encrypted data connection.

The process can be managed by a few human supervisors, who can monitor and control drone flights as well as take actions based on the information the drones collect.  

Gronstedt says a drone can inspect a crane 70% faster than a human inspector; a human inspection would tale roughly two and a half days. Drones can complete the entire inspection, and, with a bigger area covered in a single image, the drones can capture fewer images but still complete the mission, meaning they can do more work in the available flight times between battery changes. 

 Also, by being in control of the end-to-end inspection process, HHLA is able to maintain a cybersecurity safety zone around its drone operation. In fact, the entire system—the ICC, drones, and data channels—is cybersecurity certified in accordance with the standard IEC 62443.  

Challenges 

There were challenges in putting together an inspection system using drones that could successfully gather the data needed. In this case, HHLA initially found that the cameras with which the commercial drones were initially equipped could not capture images clear enough and detailed enough for a crane inspection unless the drone flew very close to the crane. But, because the cranes are massive steel structures, they distort and disrupt the communication signals required to communicate with and operate the drone if it flies too close. 

The team realized they needed cameras that were not only robust, but capable of capturing ultra high-resolution images from a greater distance, which is why they chose the Sony A7RIV and A7R V cameras, Gronstedt says.  

Also, the cameras are typically controlled manually, which meant HHLA Sky had to figure out how to automatically transfer the images from the camera to the back-end system. To accomplish this, HHLA Sky modified the source code for the camera’s SDK , Gronstedt says.  

As HHLA not only operates terminals out of Hamburg but also ports in other European cities, the company consolidated management operations, including automated crane inspection, achieving economies of scale.  

“Automated drone inspection and measurement are poised for rapid growth,” Gronstedt says. “With dramatic cost savings, end-to-end security, and increasingly accurate results, it will play a significant part in modern commercial infrastructure, logistics and planning.”