Smart IR - AI-based infrared sensors

The shift of artificial intelligence (AI) from the cloud to the edge, i.e. to the sensors, offers major advantages for applications that rely on real-time data processing. Edge AI in close proximity to the sensor drastically reduces latency times, as less data needs to be sent to central servers via networks. It also reduces the load on the communication infrastructure and protects sensitive data, as processing takes place locally. This is particularly advantageous for energy-intensive processes such as people detection, as the processing power is provided directly at the point of use with minimal energy requirements. The integration of edge AI in sensors makes it possible to develop highly scalable and autonomous systems that can react in real time without having to rely on cloud resources.

The long-term goal of Heimann Sensor GmbH is to develop a smart edge device that combines the imaging infrared sensor based on thermopile arrays from Heimann Sensors with an innovative AI chip from Fraunhofer IPMS. This AI chip is based on in-memory computing hardware that was developed using a new type of memory technology. Modern methods for energy-efficient person detection and tracking are to be executed in real time. The current project is initially developing a demonstrator solution. This aims to validate key methodological and technical aspects of the final solution. The project is aiming for a technology readiness level (TRL) of 5-6, which corresponds to a prototype solution in a realistic application environment.

Specifically, a first version of the AI chip (from process line N of Fraunhofer IPMS) will be tested in the project. Two demonstrators are being developed for this purpose: a board with an in-memory computing memory macro based on ferroelectric field-effect transistors (FeFETs) and an FPGA implementation of the entire accelerator architecture. The FPGA serves as an interim solution to realize initial applications for object recognition and tracking and to demonstrate the connection to the thermopile array. Communication between the AI chip and the thermopile chip takes place via UDP to a sensor module provided by Heimann.