Learning 2 Sense

Our vision of jointly optimizing network and sensor systems presents four fundamental ML challenges:

1. Variable data distribution: With an optimizable sensor layout, the spatial distribution of sensor data changes continuously—networks must be able to handle irregular, evolving data.

2. Physical prior knowledge: Efficient optimization benefits from an understanding of how sensor design parameters influence the data recordings. This knowledge must be integrated into network architectures.

3. Simulation-Based Training: Since real-world sensor systems are too slow to provide real-time training data, training processes rely on simulations. ML systems must bridge domain gaps and generalize to real-world data.

4. Cost function: The formulation for joint optimization of sensor and network parameters is crucial. Dividing the data into two parts to optimize different parameters can further improve generalization ability.

The project addresses these four challenges using concrete examples from 3D microscopy and THz imaging.

This subproject investigates the joint optimization of neural networks and sensors to improve prediction accuracy, generalization ability, and robustness.

Key areas of investigation include:

• Architecture search: Developing neural architectures in the context of joint network and sensor optimization
• Adversarial training: Increasing robustness and generalization ability
• Explainability: Analysis of optimized architectural and sensor systems

The subproject “Architectures for Non-Imaging L2S Image Processing Systems” investigates machine learning approaches for the development and optimization of new sensors for visible light and THz radiation.

The goal is to create sensors optimized for specific image processing tasks that do not capture traditional images. Instead, they utilize difference signals such as spatial contrast, motion, and foveal vision—many techniques inspired by the human visual system.

CMOS image sensors have revolutionized our lives and contributed significantly to the growth of AI. However, conventional digital cameras are designed for on-screen display and are not optimized for machine learning tasks. While AI algorithms are modeled after the animal brain, the image sensor is hardly inspired by the animal eye and performs very little image processing. Digital cameras therefore remain data sources with limited information extraction capabilities.

This project is developing novel image sensor architectures in conjunction with intelligent algorithms:

• Pixel geometry and layout: Redesigned for optimal input in image processing tasks (developed by L2S collaboration partners)
• Analog signal processing: Early processing close to the pixel via mathematical operators (derivatives, convolution) and input stages of neural networks
• Efficiency gains: Reduction of computational complexity for subsequent AI software

Joint research between sensor developers and AI researchers ensures that optical and computational efficiency are maintained.

The “Learning to Sense” (L2S) research unit focuses on the joint optimization of sensor system design parameters and neural networks. This requires extensive training data for various sensor and scene configurations. Since real-world data collection is often costly or impossible, simulating the sensor data generation process is a critical success factor.

This subproject focuses on the efficient simulation of the sensor data generation process:

• Physical effects: Simulation of different wavelengths, coherent radiation, and material interactions
• Imaging methods: Synthetic (unfocused) images and sensor system design (pixel and spectral filter arrangements)
• Technical focus: Forward and differentiable simulation enables ML-based application and hardware development for arbitrary parameters

This goal is guided by the following interrelated subgoals:

• Investigating how physical knowledge of network architectures can be used to develop adaptive approaches for generating synthetic images that improve image quality and correct scene-dependent interference artifacts for coherent 3D imaging.

• Evaluating and understanding the robustness of machine learning-based segmentation of reconstructed 3D THz image data derived from sparse illumination and sensor arrangements, including differential imaging modes.

• Assessing and determining whether segmentation from raw sensor data can be achieved directly, without an intermediate step of 3D image generation via synthetic reconstruction.

• To investigate how sensor-task-dependent system optimization can fundamentally maximize imaging and recognition capabilities while minimizing hardware and data acquisition requirements.