Recognizing Real Emotions in a Virtual World

Joy, frustration and boredom: In the ELISE study, scientists from the University of Siegen used a sensor mask to examine overburdening and insufficient challenging of people in virtual reality.

360-degree images accompanied by atmospheric music. Then a bare room, in the middle of which is a circle with a black line, which can be rotated 90 degrees by pressing a button. This is followed by the skill game "Hot Wire", where you have to guide a wire loop over a curved wire without touching it - as time runs out and the difficulty level increases. These three virtual scenarios each last about five minutes and are intended to trigger the emotions of joy, boredom and frustration for the user. Whether this works and whether these emotions can also be recognized in the body data is examined within the research project ELISE at the University of Siegen, which is funded by the Federal Ministry of Education and Research.

ELISE stands for "Development of an interactive and emotion-sensitive learning system for competence development in the area of business process walkthroughs". Researchers want with the help of a special sensor mask, which specially for the project at the chair for medical informatics and Microsystementwurf of Prof. Dr. med. Rainer Brück was designed to measure individual body data - such as skin conductivity, temperature, eye movements, heart rate and brain activity. A final prototype of this mask has now been tested for the first time in a two-week emotion study in a VR environment with around 100 participants at the university's Research College (FoKoS). The goal is to generate an algorithm for pattern recognition of emotions in a virtual reality application.

A first random analysis of the data was positive from the developers' point of view. "Not only do the sensor data look good, we have also received consistently good feedback from the study participants on the VR experience. Now we hope that the data can be successfully used by the research group Pattern Recognition of Prof. Dr. Marcin Grzegorzek at the University of Lübeck, which is part of the interdisciplinary project, "said Dr. Armin Grünewald from the Siegen Chair of Medical Informatics and Microsystems Design. The common goal of the scientists is the recognition of emotions in real time. For this, the algorithm must be able to link the measured body data with the associated emotions.

The real-time recognition of the triggered emotions can be used to tailor VR learning systems on an individual basis to the users. In a sample application, which was also developed in the ELISE project, apprentices undergo the work process of a post office. Here, the individual work steps such as package acceptance, franking and sorting are virtually simulated and learned. Through emotion recognition, the system reacts specifically to the individual. If the use of the sensors detects boredom in one person, the work steps are carried out faster and the learning level is increased. Conversely, work steps can be simplified and repeated until a positive mood is evident in the person.