Human computing and machine understanding of human behavior

Proceedings of the 8th …, 2006

A widely accepted prediction is that computing will move to the background, weaving itself into the fabric of our everyday living spaces and projecting the human user into the foreground. If this prediction is to come true, then next generation computing, which we will call human computing, should be about anticipatory user interfaces that should be human-centered, built for humans based on human models. They should transcend the traditional keyboard and mouse to include natural, human-like interactive functions including understanding and emulating certain human behaviors such as affective and social signaling. This article discusses a number of components of human behavior, how they might be integrated into computers, and how far we are from realizing the front end of human computing, that is, how far are we from enabling computers to understand human behavior.

Proceedings of the 8th international conference on Multimodal interfaces - ICMI '06, 2006

Many people believe that emotions and subjective feelings are one and the same and that a goal of human-centered computing is emotion recognition. The first belief is outdated; the second mistaken. For human-centered computing to succeed, a different way of thinking is needed. Emotions are species-typical patterns that evolved because of their value in addressing fundamental life tasks[19]. Emotions consist of multiple components that may include intentions, action tendencies, appraisals, other cognitions, central and peripheral changes in physiology, and subjective feelings. Emotions are not directly observable, but are inferred from expressive behavior, self-report, physiological indicators, and context. I focus on expressive behavior because of its coherence with other indicators and the depth of research on the facial expression of emotion in behavioral and computer science. In this paper, among the topics I include are approaches to measurement, timing or dynamics, individual differences, dyadic interaction, and inference. I propose that design and implementation of perceptual user interfaces may be better informed by considering the complexity of emotion, its various indicators, measurement, individual differences, dyadic interaction, and problems of inference.

IBM Systems Journal, 2000

For a long time emotions have been kept out of the deliberate tools of science; scientists have expressed emotion, but no tools could sense and respond to their affective information. This paper highlights research at the MIT Media Laboratory aimed at giving computers the ability to comfortably sense, recognize, and respond to the human communication of emotion, especially affective states such as frustration, confusion, interest, distress, anger, and joy. Two main themes of sensing-self-report and concurrent expression-are described, together with examples of systems that give users new ways to communicate emotion to computers and, through computers, to other people. In addition to building systems that try to elicit and detect frustration, our research group has built a system that responds to user frustration in a way that appears to help alleviate it. This paper highlights applications of this research to interface design, wearable computing, entertainment, and education, and briefly presents some potential ethical concerns and how they might be addressed. N ot all computers need to "pay attention" to emotions or to have the capability to emulate emotion. Some machines are useful as rigid tools, and it is fine to keep them that way. However, there are situations in which human-computer interaction could be improved by having the computer adapt to the user, and in which communication about when, where, how, and how important it is to adapt involves the use of emotional information. Findings of Reeves and Nass at Stanford University suggest that the interaction between human and machine is largely natural and social, 1 indicating that factors important in human-human interaction are also important in human-computer interaction. In human-human interaction, it has recently been argued that skills of so-called "emotional intelligence" are more important than are traditional mathematical and verbal skills of intelligence. 2 These skills include the ability to recognize the emotions of another and to respond appropriately to these emotions. Whether or not these particular skills are more important than certain other skills will depend on the situation and goals of the user, but what is clear is that these skills are important in human-human interaction, and when they are missing, interaction is more likely to be perceived as frustrating and not very intelligent. Consider an example of human-human interaction: Suppose that a human starts to give you help at a bad time. You try ignoring, then frowning at, and then maybe glaring at him or her. The savvy human infers you do not like what just happened, ceases the interruption, notes the context, and learns from the feedback. This scenario is easily modified for human-computer interaction and is relevant to the growing use of software that is designed to interrupt a person with reminders, hints, and offers of suggestions. Now we take out the word "human" in the example and replace it with the word "computer":

2006

Any interactive software program must interpret the users' actions and come up with an appropriate response that is intelligable and meaningful to the user. In most situations, the options of the user are determined by the software and hardware and the actions that can be carried out are unambiguous. The machine knows what it should do when the user carries out an action. In most cases, the user knows what he has to do by relying on conventions which he may have learned by having had a look at the instruction manual, having them seen performed by somebody else, or which he learned by modifying a previously learned convention. Some, or most, of the times he just finds out by trial and error. In user-friendly interfaces, the user knows, without having to read extensive manuals, what is expected from him and how he can get the machine to do what he wants. An intelligent interface is so-called, because it does not assume the same kind of programming of the user by the machine, but the machine itself can figure out what the user wants and how he wants it without the user having to take all the trouble of telling it to the machine in the way the machine dictates but being able to do it in his own words. Or perhaps by not using any words at all, as the machine is able to read off the intentions of the user by observing his actions and expressions. Ideally, the machine should be able to determine what the user wants, what he expects, what he hopes will happen, and how he feels. The Human Media Interaction group is carrying out several studies that involve determining in one way or another what the mental state is of a person by analyzing various aspects of his behaviour. In developing intelligent tutoring systems, for instance, we like to know as much as possible of the mental state of the students as is useful to respond appropriately to their actions. A tutoring system should create the right conditions for the students to learn something. This means adjusting the teaching strategy or the specific instructions and feedback to the student establishing the right frame of mind for optimal learning. This not only involves determining whether the students are involved and attending, but also whether or not they are motivated and how they are responding to the mistakes they are making. Do students feel the right kind of frustration? Are they bored or challenged to do better? Are they understanding the instructions? Are things too easy for them or too difficult etcetera? These are some of the things that a skilled human instructor can deduce without effort from the students' behaviours. The question is: how can we build a machine with similar skills? Also for other applications, it is extremely valuable if one could find out more about the mental state of a person. Besides intelligent tutoring systems, we are also interested in automatically interpreting the behaviours of people engaged in human to human conversation. In the research carried out in the context of the AMI project (http://www.amiproject.org/), we are looking at people involved in meetings. Observing people and interpreting their actions to find out their intentions and motivations, what people feel and think, is needed to develop intelligently searchable multimedia recordings of the meetings. Speech recognizers can help us to produce transcripts of the meeting automatically and natural language processing, information extraction and information retrieval techniques can help us to enrich the data with tags that can be used as metadata enabling semantic access. But what we need is not just access to what was said. We also have to find out how it was intended. Only to a limited extent do the actual words that were used express what was meant. Also, we might not just be interested in what the speaker had to say, but also what the listeners thought and felt about it. Was a proposal greeted with great enthusiasm or with a fair amount of scepticism? Who agreed immediately and who didn't? For both the tutoring and the meeting case, we can deduce a lot about the intentions of students and participants in the meeting by analysing the actions they performed in the task they were given or the things they said in the meeting. However, much of the information about the mental state is not presented in these actions or the words that are spoken, but rather in the way the actions are carried out and the other behaviours that accompany them. In particular, the paraverbal and nonverbal signals may give us an indication of extra dimensions of the mental state of the users of a system or the participants interacting in a meeting or other form of conversation. Recognizing and interpreting such cues is an important research area. In order to understand the way the cues work in the actual cases that we are considering, we need to look at naturalistic data to find out what behaviours are being displayed and how they are associated with diverse functions. In the paragraphs that follow we will discuss several aspects of this undertaking, focussing on the collection and analsysis of data. Both for the tutoring case

Interacting with Computers, 2002

Prototypes of interactive computer systems have been built that can begin to detect and label aspects of human emotional expression, and that respond to users experiencing frustration and other negative emotions with emotionally supportive interactions, demonstrating components of human skills such as active listening, empathy, and sympathy. These working systems support the prediction that a computer can begin to undo some of the negative feelings it causes by helping a user manage his or her emotional state. This paper clarifies the philosophy of this new approach to human-computer interaction: deliberately recognising and responding to an individual user's emotions in ways that help users meet their needs. We define user needs in a broader perspective than has been hitherto discussed in the HCI community, to include emotional and social needs, and examine technology's emerging capability to address and support such needs. We raise and discuss potential concerns and objections regarding this technology, and describe several opportunities for future work.