»Learning to Sense«
Dr. John Meshreki is conducting an experiment at a Fourier Ptychographic Microscopy setup in the optics lab at the Center for Sensor Systems (ZESS), University of Siegen. He is a researcher at Prof. Ivo Ihrke's chair of Computational Sensorics & Communications Engineering.
The human eye is like the holy grail for Bhaskar Choubey. Choubey is Professor for Analogue Circuits and Image Sensors and one of the seven principal investigators behind the DFG research unit »Learning to Sense« at the University of Siegen’s Center for Sensor Systems (ZESS), that has been running since 2023. The image sensors in even the most expensive smartphone or camera are no match for the human eye. That's because eye and image processing in the brain interact to form an incredibly effective system. It starts with the retina. The retina is covered with thousands of photosensitive cells distributed in various ways. At the point of sharpest focus, a large number of small photosensitive cells are concentrated very closely. This produces a high image resolution when we focus on an object. The photosensitive cells are larger and less concentrated at the edge of the retina. Therefore, the image here has a lower resolution.
Nevertheless, we still perceive movement very clearly. That was a life-saver for early humans, allowing our ancestors to detect approaching predators in time to react. Today, our peripheral vision stops us from stepping into the road when a car is approaching. The photosensitive cell distribution on the retina benefits image processing by the brain enormously. Because the point of sharpest vision is very small, the brain has to process only a small volume of high-resolution data. The low-resolution information at the edge of the retina is usually less important and takes up less processing effort.
Less is more
Current optical image sensors have a completely different design. Their photosensors, or sensor pixels, are arranged in a rectangular grid with even spacing. »This design makes automated image evaluation increasingly difficult in many of today's application fields,« says Margret Keuper, Professor of Machine Learning and Computer Vision at the University of Mannheim. There is a vast number of applications with a fixed camera position that need high resolution data in only a small part of the image while still relying heavily on a larger context. Those would benefit immensely from a new chip design.
Examples are quality assurance in production plants or traffic monitoring in self driving cars. The problem is that a traditional chip captures the entire image in very high resolution, creating a vast quantity of image data. The software then has to process all of this data, even though in these cases only a small section of the image is relevant – for example, a defective component on a conveyor belt or a significant change in traffic in the periphery.
This overabundance of data is a growing problem because increasingly complex AI is used for image processing – in particular neural networks that process information in multiple steps or layers. The more image information one inputs into a neural network, the more computing power one needs and the longer it takes for the neural network to produce a result.
»There are many applications where image sensors could become significantly more effective when designed in an unconventional way that is particularly well suited for a subsequent analysis with machine learning technology,« says Michael Möller, Professor of Computer Vision. Basically, both fields have developed separately from each other: electrical engineering, which has continuously optimized conventional image sensors, and computer science, which has developed its own, distinct tools. These tools have rarely been created with the needs of the other discipline in mind. »In our Learning to Sense project, we aim for the joint optimization of the design of the sensor system as well as the computer analysis of the captured data,« says Michael Möller, who is the spokesperson of the project.
Together with Möller, the researchers behind the »Learning to Sense« project are Prof. Dr. Margret Keuper, Prof. Dr. Volker Blanz, and Prof. Dr. Andreas Kolb from the field of computer sciences, and Prof. Dr. Peter Haring Bolívar, Prof. Bhaskar Choubey, and Prof. Dr. Ivo Ihrke from the field of sensor technology at the University of Siegen. »Together with our doctoral students, we aim to design new sensor systems and machine learning methods that complement each other perfectly,« explains Möller. The groups are collaborating on the development of novel techniques to jointly optimize image sensor systems as well as machine learning approaches for analyzing the resulting data. Far beyond particular applications, the goal of their fundamental research is to establish a new paradigm: the ability to »learn« the design of future sensor systems in a similar fashion as today's artificial intelligence is »learning« to understand our world.
In this context, the research unit has already optimized CMOS sensors featuring non-uniform pixel layouts to enable more efficient automatic recognition in autonomous driving scenarios. The team has also developed conversion schemes for analog-to-digital converters (ADCs) to achieve highly efficient acquisition of smartwatch, image and video data, which can be readily interpreted by a neural network. Other developments include lenses that improve the robustness of image classification and illuminations that maximize information gain in microscopy. The unit has also created inverse simulation approaches that allow the extraction of the material properties of objects imaged using visible and non-visible light in the terahertz frequency range.
Creative work in the university's chip laboratory
The CMOS sensor laboratory was established at the University of Siegen by sensor expert Prof. Dr. Bhaskar Choubey, who moved to Siegen from the University of Oxford in 2018. »In Siegen, we have the technical equipment and the right people to allow us to design and manufacture new types of sensors,« he says. »There is a mass market for CMOS chips, and Europe needs to find its sovereignty in chip design and fabrication in the new geopolitical world order.« As part of this initiative, the University of Siegen has created INCYTE (Interdisciplinary Center for Nanoanalytics, Nanochemistry and Cyber-Physical Sensor Technologies), a center with state-of-the-art facilities to create and test micro/nano sensory systems, while also developing fabrication technology for the next generation of chips. However, designing completely novel sensors quickly becomes extremely expensive. »We want to create entirely new sensor designs for the various application fields,« says Choubey, »for example, sensors with pixels arrayed in new patterns and varied spacing – just like in the human eye. A major problem is that it would be too expensive and time-consuming to build a sensor for every new design. »That's why we have to develop algorithms to test whether a new sensor design actually works,« says Margret Keuper. Only the most promising sensor designs will actually be built.
»Learning to Sense«
Today's neural networks and other AI software are so complex that even the experts who develop them can barely comprehend how they analyze data. The AI software is fed training data such as images showing typical component defects. Over time, the neural network learns what defects look like. But its internal processes remain a closed book. As long as you feed neural networks with image information a human can perceive, it is possible in retrospect to check whether the neural network has worked correctly.
For example, whether a fault detected by the software really is a hole in a component. But it gets difficult when you design completely novel sensors that don't deliver conventional image information. In this case, neural networks can learn entirely different image characteristics which humans can't see, for example the brightness difference between adjacent pixels. »That's why we need to make sure our AI solutions produce plausible results and the algorithms really do output the information we want,« says Margret Keuper.
The »Learning to Sense« project is one of eight prestigious research units funded by the German Research Foundation (DFG) in their special initiative on artificial intelligence (AI), and is supported with over 3.3 million euros. Currently, the project is applying for an additional four years of funding to extend the 'Learning to Sense' idea to a dynamic setting. The team has been strengthened by two professors – Junior Professor Dr. Jovita Lukasik who has recently been appointed professor for Visual Computing, and Professor Dr. Kristof Van Laerhoven. Both are researchers at the University of Siegen. Together, the team aims to exploit dynamic changes of sensor systems over time. The goal is to build more versatile, effective, and efficient systems for acquiring and automatically analyzing data in specific tasks. To do this, the digitalization, readout, and preprocessing of the data, as well as the optics, and the illumination must be adapted, dynamically based on previous measurements.
This story was first published in the University of Siegen research magazine.
Author of the original text: Tim Schröder
