NeurOSmart: Development of a smart LiDAR sensor

… with integrated neuromorphic data processing picks up speed.

In the Fraunhofer lead project NeurOSmart, five institutes (ISIT, IPMS, IMS, IWU, IAIS)* are jointly researching a particularly energy-efficient and intelligent sensor for the next generation of autonomously acting systems. After just under a year of project work, the Fraunhofer researchers will present the expected sensor system and the hybrid, neuromorphic data processing pipeline at Electronica.

The current trend for complex, demanding applications, such as autonomous driving, is toward mobile supercomputers with significantly increasing energy consumption. Especially for mobile systems, this leads to shorter operating times or ranges and, according to current forecasts, will even reach the limits of global energy production in the next decades. To counteract this escalation, the Fraunhofer researchers involved in NeurOSmart are relying on decentralized intelligence that is tailored to the respective sensor and new approaches to energy-efficient data processing.

The basic idea behind the ambitious project is to integrate data processing directly into the sensor as far as possible and to become as energy-efficient as possible through coordinated components.

The desired result is a range sensor (LiDAR) that detects and classifies objects on the basis of individual pixels in a field of view of 90° x 90° at working distances of up to 12 meters. All this in real time with frame rates of over 20 Hz and with an energy consumption of a few 10 W. This system can thus be used as a safety system for collaboration between humans and industrial robots and, in the future, also for mobile applications.
This performance is made possible by a variety of innovations in data processing and sensor technology, which are combined in this project.

We would like to particularly highlight the raw data preparation and pre-processing, which provides an integrated solution for the energy-efficient computation of AI algorithms on embedded hardware in real-time via the connection of the RISC-V Manycore cluster AIRISC-N with the embedded AI framework AIfES. This way, the amount of data to be interpreted is reduced by a factor of 10 and allow to observe even only parts of the scene (region of interest), which again reduces energy consumption and increases response time.

Best of all – the Fraunhofer technologies of AIRISC Core and AI Framework AIfES are available for free via GitHub and can recently also be easily downloaded as a pre-configured system from the Intel Pathfinder Shop and integrated into own designs. In addition to image processing applications (LiDAR, thermal imaging, Visible), time series analysis (acoustic monitoring, medical technology) and AI-based predictions (predictive maintenance, failure prediction), among others, are also supported via preconfigured extensions.

Advanced data interpretation is implemented on a novel mixed-signal in-memory accelerator that outperforms current alternatives by at least two orders of magnitude with an expected energy efficiency of more than 1000 TOPS/W. At the heart of the accelerator are so-called ferroelectric field-effect transistors (FeFETs), which can both store and process information (in-memory compute). This way, the energy-intensive bottleneck in the communication between memory and computational core is resolved and complex AI processes can also be realized in mobile end devices.

The advantages of the new approach are underlined by the parallel development of particularly small and efficient models for object recognition and classification. They are specifically adapted to the sensor, to new possibilities of directly integrated electronics and to their applications. The result is fast response time, increased data protection and significant energy savings compared to the current trend of off-practice or cloud-based solutions that prefer to rely on ever larger, more energy-intensive models.

At Electronica 2022, you will find the project’s researchers at the Fraunhofer-Gesellschaft booth in Hall B4, Booth 258.

*The Fraunhofer Institutes involved in the lead project are:

Fraunhofer ISIT
Fraunhofer ISIT is the coordinator of the NeurOSmart project: research competencies include the development and integration of piezoelectric and ferroelectric materials for microelectronic and electromechanical applications. Within the project, Fraunhofer ISIT is involved in the development and fabrication of the scanning LiDAR system as well as AIScN-based ferroelectric multibit memories.

Fraunhofer IPMS
One of the primary research competences is the development of memory technologies in advanced node CMOS realizations. For this purpose, Fraunhofer IPMS has an innovative ferroelectric memory technology (FeFET). In the context of NeurOSmart, the memory emulation is used, among other things, for software blocks to control the in-memory hardware blocks and to manage data streams.

Fraunhofer IMS
The core competencies of Fraunhofer IMS are the development of embedded software and AI, Smart Sensor Systems. The institute has developed AIfES (Artificial Intelligence for Embedded Systems) – a platform-independent and continuously growing machine learning library in the C programming language. AlfES is also an important prerequisite for NeurOSmart for the development of the hardware platform for sensor and scanner control.

Fraunhofer IWU
Fraunhofer IWU’s area of expertise ranges from machine tools, forming, joining and assembly technology, precision engineering and mechatronics to digitalization in production and virtual reality in the context of mechanical engineering. As part of the project, the institute will evaluate the sensor system for applicability in an industrial environment.

Fraunhofer IAIS
Fraunhofer IAIS has research expertise in the areas of distributed learning, speech assistance systems and computer vision for autonomous driving. In this context, the institute has a powerful speech recognition system for automatic transcription and recognition of speech signals. Within the framework of NeurOSmart, the institute bundles its competencies for neural network training, among other things.

Wissenschaftliche Ansprechpartner:

Dr. Michael Mensing
michael.mensing@isit.fraunhofer.de

Weitere Informationen:

https://www.isit.fraunhofer.de/en/news/neurosmart–development-of-a-smart-lidar-…