Designing a robot that can sense human emotion

Forget the robot child in the movie “AI.” Vanderbilt researchers Nilanjan Sarkar and Craig Smith have a less romantic but more practical idea in mind.

“We are not trying to give a robot emotions. We are trying to make robots that are sensitive to our emotions,” says Smith, associate professor of psychology and human development.

Their vision, which is to create a kind of robot Friday, a personal assistant who can accurately sense the moods of its human bosses and respond appropriately, is described in the article, “Online Stress Detection using Psychophysiological Signals for Implicit Human-Robot Cooperation.” The article, which appears in the Dec. issue of the journal Robotica, also reports the initial steps that they have taken to make their vision a reality.

“Psychological research shows that a lot of our communications, human to human, are implicit,” says Sarkar, an assistant professor in mechanical engineering. “The better we know the other person the better we get at understanding the psychological state of that person. So the prime motivation of our research is to determine whether a robot can sense the psychological state of a human person. Sooner or later, robots will be everywhere. As they become increasingly common, they will need to interact with humans in a more natural fashion.” When Sarkar first approached him about collaborating on the project, Smith admits that he was very skeptical. “I expected to listen and then explain to him why his ideas would never work.” But the engineer surprised him on two counts: the amount he knew about the psychophysiology of emotions and his realization that any system for detecting emotions cannot be universal, but must be based on individual patterns.

The project has two basic parts, and both are ambitious. One is to develop a system that can accurately detect a person’s psychological state by analyzing the output of a variety of physiological sensors. The other is to process this information in real time (as it happens) and convert it into a form that a computer or robot can process.

“Psychologists have been trying to identify universal patterns of physiological response since the turn of the century without success. All this effort has shown is that there are no such universal patterns,” says Smith. “The hard fact is that different individuals express the same emotion rather differently. But I think that we have established the feasibility of the individual-specific approach that we are taking and there is a good chance that we can succeed,” says Smith.

The Vanderbilt researchers are using an approach similar to that adopted by voice and handwriting recognition systems. They are gathering baseline information about each person and analyzing it to identify the responses associated with different mental states. One advantage that the researchers have is the recent advances in sensor technology. “Extremely small, ’wearable’ sensors have been developed that are quite comfortable and are fast enough for real-time applications,” says Sarkar.

Their first experiments concentrated on detecting high and low anxiety levels using a heart rate monitor. “There are sophisticated medical diagnostic techniques that can detect stress in a patient,” they acknowledge in their Robotica paper, but add, “All those techniques are slow, expensive and, more importantly, not suitable for a person who is moving and working.”

In this case the researchers used playing video games to put subjects under pressure and induce stress. By varying the level of difficulty of the games, they were able to vary the level of stress involved. They obtained electrocardiogram data from several video-gaming playing subjects over a six-month period.

Sarkar and his research team used advanced signal processing techniques, including wavelet analysis and fuzzy logic, to analyze the heart-rate data. They looked specifically at variations in the interval between heartbeats, a common measure of heart rate variability. They identified two frequency bands that vary predictably with changes in stress levels. These bands are associated with the parasympathetic and sympathetic divisions of the autonomic nervous system. The parasympathetic system reduces heart rate and tends to control heart rate under most normal conditions. The sympathetic system responds to fear and excitement and tends to increase heart rate during emergency situations.

“In all the experiments we conducted, we found that, when a subject became stressed, the level of sympathetic activity increased and level of parasympathetic activity decreased,” Sarkar says.

He and his research team have since supplemented their measures of heart rate with measures of skin conductance (affected by variations hand sweating) and facial muscle activity (brow furrowing and jaw clenching). They were able to combine this information to produce a series of rules that allow a robot to respond to information about a person’s emotional state. They have used these to program a small mobile robot. The robot is initially given a task of exploring the room. So it begins moving randomly about on the floor. Then physiological data of a person experiencing high anxiety levels is sent to a processor that detects the anxiety level and instructs the mobile robot to move to a specific location and say, “I sense that you are anxious. Is there anything I can do to help?”

In order to investigate additional psychological states, Smith has created three simple tasks – anagram, sound discrimination and math problems that systematically increase difficulty – that are designed specifically to make the performer frustrated or bored. They will be adding additional sensors, such as electroencephalogram (EEG) brain wave monitors and additional measures of cardiovascular activity. The next challenge that the researchers face is finding a way to discriminate between high levels of anxiety and engagement. These two states are accompanied by physiological responses that are much closer to each other than either of them are to low levels of anxiety or engagement. “This is the really big one,” says Smith.

The research is supported by grants from the National Science Foundation, the NASA Institute for Advanced Concepts and Vanderbilt University.