Innovations in Crossing the Line

Bo Ling, Migma Systems

Pedestrian detection at street intersections remains a challenging task even after decades of research. Current technologies for pedestrian detection such as microwave radar, ultrasonic, acoustic, video image processing, passive and active infrared, piezoelectric, and magnetic sensors are quite ineffective because of the highly dynamic backgrounds at outdoor street intersections.

Migma Systems has developed a stereo vision-based technology for pedestrian detection at traffic intersections under US Federal Highway Administration SBIR Phase I/II funding. The prototype IR LED wireless stereo vision system and advanced pedestrian detection algorithms can detect pedestrians both day and night under all weather conditions. Generic 3-D features extracted from the stereo disparity map generated by the system are used to discriminate pedestrians from vehicles since vehicle bodies are basically flat and human bodies have concave shapes.

Migma’s stereo vision system is composed of two high-resolution, 1/3-in., color CCD sensors and lenses each with 100 IR LED emitters. The camera system provides high-resolution color images during the daytime and black-and-white pictures in lower light conditions such as evening and night. The LED emitters operate at 850 nm and make it possible to detect pedestrians up to 80 ft from the camera in a totally dark environment. Migma has developed a prototype of this stereo vision system (see Fig. 1) and is currently developing a commercial product.

FIGURE 1. A pedestrian-detection stereo vision system developed by Migma Systems includes two high-resolution, 1/3-in., color CCD sensors and lenses each with 100 IR LED emitters. The camera system provides high-resolution color images during the daytime and black-and-white pictures in lower light conditions.

null

Intelligent traffic light management

Several thousand pedestrians are killed at street intersections in the US each year. The pedestrian detection system can be used to prevent the collision of pedestrians and vehicles at these intersections and is a viable commercial technology for street intersection traffic light management.

At standard street intersections, pedestrian walk signals run for a fixed time. To request the walk service at many of these intersections, pedestrians push a button on the traffic pole. Once the service request is granted their light changes from red to green and they can go across the crosswalk. However, in many cases, after pressing the button, pedestrians may simply cross the street if they do not see any approaching vehicles. When the crosswalk light finally turns to green, pedestrians are no longer occupying the crosswalk and vehicles are still stopped in all directions.

Sometimes pedestrians take a very short time to walk across the intersection and stopped vehicles still cannot move before the service request time has expired. However, slower-walking pedestrians such as senior citizens or people with disabilities may need service time longer than the standard programmed service time. Therefore, it is desirable to reduce or extend this time based on the occupancy of the crosswalk.

These two applications—reducing or extending crosswalk service time—require a pedestrian monitoring system installed at the street intersection. When pedestrians are detected walking in the crosswalk, a signal from the detection system is sent to the traffic controller, which in turn keeps the intersection traffic light phase unchanged. As long as pedestrians occupy the crosswalk, the traffic lights remain red to stop vehicles in all directions. When the crosswalk has been unoccupied for a short time, the system can send another signal to the traffic controller, which in turn changes the traffic light phase and allows the flow of vehicles (see Fig. 2).

FIGURE 2. An intelligent traffic-light management system can be installed at an intersection based on a pedestrian detection system developed by Migma Systems. The IR LED stereo vision system detects pedestrians and sends wireless signals to the traffic controller.

The system can reduce traffic congestion at street intersections, ease the frustration of drivers, reduce gasoline consumption, and reduce pollution, while increasing safety.

Pedestrian detection

Software algorithms are the key in developing the stereo vision pedestrian detection system. The most important issue for such a system is the computation time: approximately four to five detections per second must be made to ensure the early detection of pedestrians. All the algorithms are implemented in C/C++.

Images are acquired from the right and left cameras in the stereo vision system in a sequential fashion: A new image is captured by one camera once an image by the other camera has been completely processed. Thus two consecutive images (right or left) are taken from the vision system and a difference image is made by subtracting these two images. To overcome problems associated with less-than-ideal difference images, a suite of methods has been developed to filter out noise generated from the cameras, camera jiggling, and illumination changes.

One of the main advantages of using a stereo vision system is the ability to relate the distance between object and cameras and the disparity obtained from the two images taken by the cameras. However, to accomplish the goal of four to five detections per second, a computation-intensive disparity map estimation scheme is not acceptable. Consequently, Migma developed a disparity map estimation method that satisfies the time constraint for near real-time pedestrian detection (see Fig. 3).

FIGURE 3. In the pedestrian detection system, two consecutive images (right or left) are taken from the vision system and a difference image is made by subtraction. A suite of algorithms filters out noise generated from cameras, camera jiggling, and illumination changes. A classifier fusion algorithm incorporates the detections at both pixel level and windowed object level.

Ideally, the disparity map of a pedestrian has the corresponding body parts of the pedestrian such as head, upper body, and legs. However, because of camera noise, jiggling, and illumination changes, patches in the disparity map associated with a pedestrian are often disconnected or sometimes incomplete, and it is difficult to extract human body parts from the disparity map.

Other capabilities of the system include IP wireless communication, image-based camera calibration, and interface with traffic controllers. The vision system is IP enabled and can be accessed over the Internet. A built-in web server allows users to connect and view live images from traffic intersections. A software driver was developed to adjust the alignment of the two camera lenses, and it can be used in handheld computers for system configuration. Finally, the detection system can communicate with different traffic controllers to intelligently manage the traffic light phases at the street intersection.

Field testing

With the help from Massachusetts Highway Engineering Department (District 2), a prototype system was installed at a state highway intersection. This unattended field trial lasted more than three weeks, 24 hours per day. Since the wireless transceivers could not be placed inside the metallic traffic controller cabinet, they were placed in a wooden box mounted to the side of the cabinet. The stereo vision system was mounted on top of a traffic pole. A separate underground power cable was connected to the stereo system; it passed inside the traffic light pole and was essentially invisible to the public.

This field trial system was of a standalone system that can recover from power failure and Internet communication failure. Every 5 minutes, the system sent a current snapshot of the crosswalk to a web server to monitor the operating status of the system. Since it was an unattended field trial, the detection results and images were also sent to a web server, which was monitored by several groups of researchers, including the federal and state highway departments. Live detection results could be viewed to ensure that the test data were not altered.

Many individuals and groups of pedestrians were detected during the 3-week field trial, including slow-walking pedestrians, bicyclists, pedestrians walking at night, and pedestrians with long shadows (see Fig. 4). During the field trial, one camera of the stereo vision system recorded images every 2 s during four 2-hr periods of a day:

The images were then used to estimate the false alarm rate and missing detection rate.

Based on analysis of these images, the pedestrian detection system exhibited a false alarm rate at 1 per 60 to 70 minutes and a missed-pedestrian rate of 1 per day in a continuous operation setting. For detection applications in pushbutton-triggered pedestrian intersections, the system is expected to attain a 100% detection rate and 0% false alarm rate. Migma is currently developing a commercial product that features the smart IR LED IP stereo vision system with all of the detection algorithms embedded.

Bo Ling is president and CEO at Migma Systems (Walpole, MA, USA; www.migmasys.com).