Intelligent machines are increasingly becoming part of our everyday lives. Obvious examples are autonomous driving or modern robot vacuum cleaners in the home. But even where we do not come into direct contact with them, independently working robots are increasingly indispensable: as nimble and tireless helpers in logistics centers or as precise and strong partners in manufacturing environments.

In order to be able to work autonomously in these applications, i.e. as independently as possible without supervision by humans, these robots are peppered with sensors and electronics that enable them to perceive their environment and also cope with unforeseen situations independently. The more complicated the task, the more intelligent and agile the machine needs to be. As a result, more and more and different sensors are being combined, for example for distance measurement, for motion detection or for pressure determination in the event of contact, and the electronics and computer technology for recording and processing the data must be increasingly powerful.

However, this trend toward mobile supercomputers is accompanied by a significant increase in energy consumption. Particularly in the case of mobile systems, this leads to shorter operating times or ranges and, according to current forecasts, will even push the limits of global energy production in the next few decades.

To counteract this escalation, the Fraunhofer researchers involved in NeurOSmart under the leadership of the Fraunhofer ISIT are relying on a neuromorphic in-memory accelerator, which is tailored to the respective sensor. The neuromorphic electronics to be developed are modeled on the human brain, which is very energy-efficient when making decisions despite its enormous computing power.

„This kind of data processing, i.e. thinking, is realized by a novel analog computer memory technology that is also capable of performing computational operations when data is newly acquired in the system,“ explains ISIT scientist and project leader Dr. Michael Mensing: "In practice, this is used to recognize objects and their behavior accurately and in real time." Until now, this mode of operation has required several separately developed components in computers and particularly energy-intensive communication between them.

The advantages of the new approach are underlined by the parallel development of particularly small and efficient models for object recognition and classification, which are specially adapted to the sensor, the new possibilities of directly integrated electronics and their applications. The result is fast response time, increased data protection and significant energy savings compared to the current trend of off-practice or cloud-based solutions that prefer to rely on ever larger, more energy-intensive models.

In the project period of four years with a financial volume of eight million euros, this approach is to be combined for the first time with a complex LiDAR system developed at Fraunhofer and tested in an application-oriented environment. This sensor system is a crucial component of autonomously operating systems, as it uses detailed distance information to recognize its surroundings even in bad weather and over a wide range of distances. As a first sample of the novel sensors, they will be integrated in the next few years into robotic systems that support their human colleagues in manufacturing environments, for example by moving heavy loads or reaching components.

Within the project, Fraunhofer IPMS is developing the circuitry of the neuromorphic accelerator. The core of the analog accelerator is formed by modern HfO2-based crossbars. This allows the complex, often iterative digital computations of AI models (e.g. in CNNs), to be mapped to energy-efficient memory operations in the circuit. Otherwise they would have to be computed with high energy consumption on generalized graphics cards. For internal control of data flows, the RISC-V processor core EMSA5 is implemented with direct interfaces to the analog accelerator board and higher-level systems, as well as error protection mechanisms. Research into a software-programmable system architecture will enable flexible use of the circuit in a wide range of applications.

Furthermore, Fraunhofer IPMS is training the circuit of the neuromorphic accelerator together with the Fraunhofer IAIS. In a first step, the neural network model for LiDAR data evaluation is being researched. The goal here is the automated mapping on the available hardware topology as well as its transfer to a circuit design. In the system, the circuit analyzes preprocessed LiDAR data. Thereby, the communication between pre-processing (FPGA) and accelerator circuitry takes place via a real-time capable high-speed Ethernet interface.

Finally, Fraunhofer IPMS is involved in the validation of the new memory cell approaches for future accelerator implementations together with the Fraunhofer Institute for Silicon Technology ISIT.

About Fraunhofer IPMS

The Fraunhofer Institute for Photonic Microsystems IPMS stands for applied research and development in the fields of intelligent industrial solutions, medical technology and improved quality of life. Our research focuses on miniaturized sensors and actuators, integrated circuits, wireless and wired data communication, and customized MEMS systems.