Engineers Use Deep Learning to Develop New Embedded Vision Applications
Applications based on algorithms developed with deep learning are fueling at least part of the continued growth in embedded vision, experts say.
The overall market for embedded vision—including both industrial and consumer applications—is forecasted to expand at a rate of 7.7% between 2022-2032, reaching $159 billion by 2031, according to Transparency Market Research (www.transparencymarketresearch.com; Wilmington, DE, USA).
Embedded vision is typically defined as image capture and processing—and often involves analysis with intelligent vision algorithms—running on the same piece of hardware. The goal is to send results to a computer or the cloud for further analysis or storage rather than sending data on every pixel. Often the sensor and image processing board are optimized for a specific application. “You’re sending conclusions—not the whole thing,” explains Matthew Dyson, Principal Technology Analyst at IDTechEx (Cambridge, UK;www.idtechex.com/).
By reducing data transfer requirements, data processing is faster, requires less power and is less expensive, particularly if cloud space is rented, such as is the case for many software-as-a-service (SaaS) solutions.
“Embedded vision describes an approach of combining existing capabilities for additional benefits,” Dyson explains.
“To me, embedded vision would be a subset of machine vision, and it means I'm doing the machine vision but in situ,” he adds.
The applications for which embedded vision makes sense are those in which you plan to ask a limited and consistent set of questions and where low latency is important. For example, industrial inline inspection involving the identification of shapes or product orientation. “I think you could apply it anywhere that you’re inspecting similar things or performing similar tasks,” Dyson says.
The Role of Deep Learning
Deep Learning typically is a subfield of machine learning involving multiple neural networks. That is why deep learning requires significant computational power, memory, and storage. This makes using deep learning on edge devices somewhat problematic, as embedded devices are designed for applications requiring fewer resources, explains David Wyatt, Founder of Automation Doctor and Vision System Doctor (Jacksonville, FL, USA; www.automation-doctor.com).
“So how do we get the advantages of deep learning to these front-line troops? The first solution is to not do your deep learning at the edge. Use the edge devices as swarms to collect data for deep learning elsewhere. These results can then be pruned or distilled and squeezed to make a smaller net that can fit at the edge,” Wyatt says.
That is what is happening today and is why embedded vision applications involving AI tend to be designed to tackle a very specific problem in inspection or automation.
Jeff Bier, Founder of the Edge AI and Vision Alliance (Walnut Creek, CA, USA; www.edge-ai-vision.com), which organizes an annual summit on the topic, explains further. “There are literally thousands of new embedded vision applications powered by deep learning, spanning virtually every industry,” he says.
In the organization’s most recent survey of computer-vision developers, 83% of respondents said that they’re using deep learning in embedded vision-based products.
Applications at the Edge
Examples include autonomous robots and drones in which rapid and repetitive answers to the same vision questions, such as those involving obstacle identification, are important.
On a factory floor or warehouse, for example, robots that work alongside people, known as co-bots, rely on route planning and obstacle avoidance. This often involves many redundant sensors—such as LiDAR, ultrasonic, and 2D and 3D cameras—in combination with deep learning-enabled algorithms, Dyson explains. “They, obviously, need to have a good sense of awareness as to where they are, where their human compatriots are, and what work people are doing,” he says.
Another nascent application is an autonomous drone, which is more of a future use case. It would likely rely on embedded vision applications to fly without human assistance. “There won’t be an operator, so they need to be able to identify trees, hazards, birds, other drones, power lines and weather—and all of this stuff by themselves,” Dyson says. “They probably don’t want to send all of this data up to the cloud to get processed. They want to extract the conclusions really quickly in a small compact device.”
When it comes to deep learning specifically, Bier notes that developers are applying deep learning to the development of devices to monitor chronic medical conditions. Verily Life Sciences (San Francisco, CA, USA; https://verily.com), subsidiary of Alphabet, has created algorithms that complement a biosensing watch that the company created. The watch uses sensors to collect biometric health information, such as heart rate or sweating. By applying deep learning-enabled algorithms to the biometric information, Verily’s researchers hope to predict a change in the medical status of people with chronic medical conditions such as PTSD and depression or Parkinson’s disease.
Wyatt says vision applications at the edge will evolve and become more complex as edge-computing devices are designed to support them. “I think at the end of the day, we will see smarter and stronger edge devices. There is always room for a breakthrough in processor designs,” he says. this would increase the resources available at the edge for neural networks.
“This growth may start to blur the lines to the point that an edge station looks more and more like a well-connected classical vision system with intelligence,” Wyatt says.
Cameras, Sensors, and Software
Vendors are responding to the growth in demand for edge applications with cameras, sensors, and other enabling products.
As Bier explains, “Because of the vast number of new applications being developed or planned, technology suppliers are investing in a wide variety of building-block components. One area of vigorous innovation is edge processors that can run demanding deep learning algorithms with very modest cost and power consumption. Big investments have also been made in improved development tools, which make it easier for product designers to incorporate deep learning into their designs without having to master all of the details of the associated algorithms and code.”
Vision Systems Design includes information on some product offerings below.
For example, e-con Systems (Riverside, CA, USA; www.e-consystems.com) recently launched the DepthVista MIPI_IRD, a time-of-flight (ToF) camera, which is designed for both indoor and outdoor embedded vision systems. The camera has a scalable depth range of up to 12 m (6 m by default). The chroma camera uses the MIPI CSI 2 interface to connect to NVIDIA Jetson AGX Orin/ AGX Xavier. The camera supports infrared wavelengths of 940 nm and 850 nm. The camera has a resolution of 640 x 480 at 30 fps, pixel size of 5.6 x 5.6 μm, and sensor active area of 640 H x 480 V.
For its part, Imaging Development Systems (IDS) (Obersulm, Germany; www.ids-imaging.com) released IDS NXT 3.0 software for its IDS NXT cameras. The update allows the cameras to detect anomalies independently using AI.
An advantage of the updated software: engineers only need to use “good” images for training, speeding the development of new applications. IDS also improved transparency in the training process with maps that illustrate the areas in an image impacting classification results. “In this way, users can identify and eliminate training errors before a neural network is deployed in the cameras,” IDS explains.
Another update in the IDS software is a code reading function in the block-based editor, which enables the cameras to locate, identify, and read different types of code.
Basler (Ahrensburg, Germany; www.baslerweb.com) released 12 new models of the Basler dart camera, which are compatible with many embedded processing platforms. The cameras also come in CS-Mount and bare board versions.
Available in monochrome and color versions, the camera models integrate Pregius IMX287 and IMX273 sensors. They achieve frame rates of 525 fps at VGA, and 227 fps at 1.6 MP resolution. The cameras are compatible with USB 3.0 and BCON for MIPI interfaces. They are compact and low weight; the bare mount version measures 27 x 27 mm and weighs 5 g.
They are available with embedded vision add-on kits that include cable, lens, and adapter.
Rhonda Software (Chicago, IL, USA; www.rhondasoftware.com) markets the ADAS camera EVK, an off-the-self product suited for the development of driving safety systems. The camera is based on the Ambarella CV25 edge AI vision System on Chip (SoC). The camera is a joint effort with Brodmann17, a provider of Vision-AI technology for automated driving.
Brodmann17's software saves on computing power, which allows for real-time, edge video processing on-camera. The platform provides predictive notifications on road events, gathered by the camera, and detected by the ADAS perception software. The ADAS set includes forward collision warning and lane keeping assistance as well as pedestrian and vehicle detection.
Meanwhile, Teledyne e2v (https://imaging.teledyne-e2v.com) last fall announced its 2-MegaPixel Optimom camera. The Optimom 2M features a native MIPI CSI-2 protocol and standard FPC connector to link with embedded processing boards. With a 25 mm square design, the camera can fit into constrained mechanical systems. With monochrome and RGB options, the camera is fitted with Teledyne e2v’s proprietary 2-Megapixel global shutter image sensor. It has the option of being provided with an on-demand lens, which is already installed and focused. Compatible with M12 (S-Mount) lenses. Resolution of 1920 x 800.
MATRIX VISION GmbH (Oppenweiler, Germany; www.matrix-vision.com) also released new camera models for embedded vision applications. The mvBlueNAOS4 is designed for the Sony Preguis S sensor. GenICam-compatible software support ensures compatibility with existing image processing programs. Resolution for the new mvBlueNAOS4 models range from 5.1 to 24.6 MPixel. The cameras use PCI Express for image transmission, allowing direct access to the memory and processor in computers and embedded processor boards.
What to learn about these or other products related to embedded vision? Visit Vision Systems Design’s online Buyers Guide: www.vision-systems.com/directory.
Linda Wilson | Editor in Chief
Linda Wilson joined the team at Vision Systems Design in 2022. She has more than 25 years of experience in B2B publishing and has written for numerous publications, including Modern Healthcare, InformationWeek, Computerworld, Health Data Management, and many others. Before joining VSD, she was the senior editor at Medical Laboratory Observer, a sister publication to VSD.