SCIENTIFIC RESEARCH: Spatio-temporal imaging mimics animal behavior

Study of nocturnal animals by researchers at Lund University (Lund, Sweden; www.lu.se/lund-university) has led to the development of sophisticated algorithms for noise reduction and contrast enhancement in noisy image data with low dynamic range.

Using these algorithms, biology professor Eric Warrant and mathematicians Henrik Malm and Magnus Oscarsson are developing a color camera for low-light-level applications. According to Malm, the technology will be commercialized by Nocturnal Vision (www.nocturnalvision.se), which is currently licensing the technology for customized applications.

Spatio-temporal filtering combined with contrast-limited histogram equalization allows low-light, noisy images to be sharpened and their contrast improved. To see a video of the algorithm in action, go to www.nocturnalvision.se/Nocturnal_Vision.html.

In nocturnal animals, visual signals are summed both in space and time to increase their signal strength, the relative amount of spatial and temporal summation depending on the type of animal. In nocturnal animals that move quickly and need to react quickly, spatial summation is favored over temporal summation. In nocturnal animals that move slowly, the opposite is true.

To emulate this process in software, Malm and Oscarsson have developed a series of algorithms that process noisy video image data with low dynamic range. Rather than take a per-frame spatial approach, the algorithm uses a spatio-temporal method in which each pixel in a series of video frames is processed by a kernel calculated by analyzing adjacent pixels both spatially and temporally.

Because these kernels must be large when images of uniform intensity are processed and small when important structural high-frequency edges must be processed, the intensity structure within spatio-temporal images first must be processed to find these pixel intensity directions. After finding the 3-D intensity directions, the system can then dynamically adapt the filter kernel depending on the value returned.

To analyze these directions, a second moment matrix or structure tensor is constructed from the differential gradient of the image in 3-D space. The magnitude of each eigenvalue of this structure tensor provides a measure of the amount of intensity variation in the direction of the corresponding eigenvalue in 3-D space.

These values are then used to construct a matrix kernel that can be applied locally to each pixel in the image to reduce image noise. After reducing this noise, pixel intensities are transformed using contrast-limited histogram equalization to spread the intensities more uniformly and widen the dynamic range of darker areas within the image.

To demonstrate the effectiveness of this technique, Warrant, Malm, and Oscarsson first captured a sequence of image data using a Sony DCR-HC23 handheld consumer camera. Captured 360 × 288-pixel image sequences from the camera were processed using a CPU equipped with a GeForce 8800GTX graphics processor card from Nvidia (see figure).

The CPU was used to perform preprocessing stages such as image input, output, and contrast-limited histogram equalization, since such functions cannot easily be partitioned on a CPU. However, the most computationally expensive functions such as the calculation of structure tensor, gradient calculation, and filtering were partitioned to run on the GPU.

A number of frames were first uploaded to the graphics card as floating-point 2-D textures to calculate the spatial gradients. Then, after temporal differences of each successive frame were computed, the resulting gradient was used to compute the structure tensor for each pixel in each frame. After computing the tensor for each frame, eigenvalues and eigenvectors of the structure tensor were computed and used to generate the filter weights of each kernel.