This paper gives an overview of current research on pedagogical agents at CARTE in the USC / Info... more This paper gives an overview of current research on pedagogical agents at CARTE in the USC / Information Sciences Institute. Pedagogical agents, also nicknamed "guidebots," interact with learners in order to help keep learning activities on track. Social intelligence is required in order to engage in tutorial interactions with the learners, properly interpret the interactions of learners with other agents, and to model characters that can interact with the learners in social settings. At the same time, depth of domain intelligence is typically required in order for the agents to work effectively as advisors or collaborators.
Pedagogical agent research seeks to exploit Reeves and Nass's Media Equation, which holds that us... more Pedagogical agent research seeks to exploit Reeves and Nass's Media Equation, which holds that users respond to interactive media as if they were social actors. Investigations have tended to focus on the media used to realize the pedagogical agent, e.g., the use of animated talking heads and voices, and the results have been mixed. This paper focuses instead on the manner in which a pedagogical agent communicates with learners, on the extent to which it exhibits social intelligence. A model of socially intelligent tutorial dialog was developed based on politeness theory, and implemented in an agent interface. A series of Wizard-of-Oz studies were conducted in which subjects either received polite tutorial feedback that promotes learner face and mitigates face threat, or received direct feedback that disregarded learner face. The polite version yielded better learning outcomes, and the effect was amplified in learners who expressed a preference for indirect feedback. These results confirm the hypothesis that learners tend to respond to pedagogical agents as social actors, and suggest that research should perhaps focus less on the media in which agents are realized, and place more emphasis on the agents' social intelligence.
Back in the 1990s, we started work on pedagogical agents — a novel paradigm for interactive learn... more Back in the 1990s, we started work on pedagogical agents — a novel paradigm for interactive learning. Pedagogical agents are autonomous characters that inhabit learning environments to engage with learners in rich, face‐to‐face interactions. Building on this work, in 2000, together with our colleague Jeff Rickel, we published an article on pedagogical agents (Johnson, Rickel, and Lester 2000) that surveyed and discussed the potential of this new paradigm. We made the case that pedagogical agents that interact with learners in natural, lifelike ways can help learning environments achieve improved learning outcomes. This article has been widely cited, and was a winner of the 2017 IFAAMAS Award for Influential Papers in Autonomous Agents and Multiagent Systems.1On the occasion of receiving the IFAAMAS award, and after 20 years of work on pedagogical agents, we take another look at the future of the field. We start by revisiting the predictions we made in 2000 for pedagogical agents, an...
Virtual reality can broaden the types of interaction between students and computer tutors. As in ... more Virtual reality can broaden the types of interaction between students and computer tutors. As in con- ventional simulation-based training, the computer can watch students practice tasks, responding to questions and offering advice. However, immersive virtual envi- ronments also allow the computer tutor to physically inhabit the virtual world with the student. Such a "pedagogical agent" can physically collaborate with the
We are working towards animated agents that can carry on tutorial, task-oriented dialogs with hum... more We are working towards animated agents that can carry on tutorial, task-oriented dialogs with human students. The agent's objective is to help students learn to perform physical, procedural tasks, such as operating and maintaining equipment. Although most research on such dialogs has focused on verbal com- munication, nonverbal communication can play many important roles as well. To allow a wide
This paper describes the potential for instructing animated agents through collaborative dialog i... more This paper describes the potential for instructing animated agents through collaborative dialog in a simulated environment. The abilities to share activity with a human instructor and employ both verbal and nonverbal modes of communication allow the agent to be taught in a manner natural for the instructor. A system that uses such shared activity and instruction to acquire the knowledge needed by an agent is described. This system is implemented in STEVE, an embodied agent that teaches physical tasks to human students.
Proceedings of the fifth international conference on Autonomous agents, 2001
We describe an initial prototype of a holodeck-like environment that we have created for the Miss... more We describe an initial prototype of a holodeck-like environment that we have created for the Mission Rehearsal Exercise Project. The goal of the project is to create an experience learning system where the participants are immersed in an environment where they can encounter the sights, sounds, and circumstances of realworld scenarios. Virtual humans act as characters and coaches in an interactive story with pedagogical goals.
Proceedings of the 8th international conference on Intelligent user interfaces - IUI '03, 2003
Animated pedagogical agents offer promise as a means of making computer-aided learning more engag... more Animated pedagogical agents offer promise as a means of making computer-aided learning more engaging and effective. To achieve this, an agent must be able to interact with the learner in a manner that appears believable, and that furthers the pedagogical goals of the learning environment. In this paper we describe how the user interaction model of one pedagogical agent evolved through an iterative process of design and user testing. The pedagogical agent Adele assists students as they assess and diagnose medical and dental patients in clinical settings. We describe the results of, and our responses to, three studies of Adele, involving over two hundred and fifty medical and dental students over five years, that have led to an improved tutoring strategy, and discuss the interaction possibilities of two different reasoning engines. With the benefit of hindsight, the paper articulates the principles that govern effective user-agent interaction in educational contexts, and describes how the agent's interaction design in its current form embodies those principles.
Proceedings of 2002 IEEE Workshop on Speech Synthesis, 2002.
Text-to-speech synthesis can play an important role in interactive education and training applica... more Text-to-speech synthesis can play an important role in interactive education and training applications, as voices for animated agents. Such agents need high-quality voices capable of expressing intent and emotion. This paper presents preliminary results in an effort aimed at synthesizing expressive military speech for training applications. Such speech has acoustic and prosodic characteristics that can differ markedly from ordinary conversational speech. A limited domain synthesis approach is used employing samples of expressive speech, classified according to speaking style. The resulting synthesizer was tested both in isolation and in the context of a virtual reality training scenario with animated characters.
Human teaching strategies are usually inferred from transcripts of face-to-face conversations or ... more Human teaching strategies are usually inferred from transcripts of face-to-face conversations or computer-mediated dialogs between learner and tutor. However, during natural interactions there are no constraints on the human tutor's behavior and thus tutorial strategies are difficult to analyze and reproduce in a computational model. To overcome this problem, we have realized a Wizard of Oz interface, which by constraining the tutor's interaction makes explicit his decisions about why, how, and when to assist the student in a computer-based learning environment. These decisions automatically generate natural language utterances of different types according to two "politeness" strategies. We have successfully used the interface to model tutorial strategies.
This paper describes a pedagogical agent that helps children to learn to author structured presen... more This paper describes a pedagogical agent that helps children to learn to author structured presentations about explanations of concepts. Using a Rhetorical Structure Theory analysis of a source Web page, the agent performs pedagogical tasks to support the user's understanding of rhetorical relations, stimulates reflection about the relations between the structure of the original text and the structure of the presentations, and suggests ways to improve the user's performance. Upon completion of the authoring, the presentations are organized into coherent structures that can be performed by animated characters, or Digital Puppets, in a learning-by-teaching classroom context.
Virtual reality can broaden the types of interaction between students and computer tutors. As in ... more Virtual reality can broaden the types of interaction between students and computer tutors. As in conventional simulation-based training, the computer can watch students practice tasks, responding to questions and offering advice. However, immersive virtual environments also allow the computer tutor to inhabit the virtual world with the student. Unlike previous, disembodied computer tutors, such a "pedagogical agent" can "physically" collaborate with students, enabling new types of interaction. To illustrate the possibilities, this paper describes Steve, a pedagogical agent for virtual environments that helps students learn procedural tasks. After providing an overview of Steve's capabilities, the paper focuses on the benefits and challenges of graphically representing Steve in the virtual environment.
Proceedings of the fourth international conference on Autonomous agents, 2000
This paper describes an agent-based approach to realizing interactive pedagogical drama. Characte... more This paper describes an agent-based approach to realizing interactive pedagogical drama. Characters choose their actions autonomously, while director and cinematographer agents manage the action and its presentation in order to maintain story structure, achieve pedagogical goals, and present the dynamic story to as to achieve the best dramatic effect. Artistic standards must be maintained while permitting substantial variability in story scenario. To achieve these objectives, scripted dialog is deconstructed into elements that are portrayed by agents with emotion models. Learners influence how the drama unfolds by controlling the intentions of one or more characters, who then behave in accordance with those intentions. Interactions between characters create opportunities to move the story in pedagogically useful directions, which the automated director exploits. This approach is realized in the multimedia title Carmen's Bright IDEAS, an interactive health intervention designed to improve the problem solving skills of mothers of pediatric cancer patients.
This paper describes a software agent that learns procedural knowledge from a human instructor we... more This paper describes a software agent that learns procedural knowledge from a human instructor well enough to teach human students. In order to teach, the agent needs more than the ability to perform a procedure. It must also be able to monitor human students performing the procedure and be able to articulate the reasons why actions are necessary. Our research concentrates on helping an instructor instruct the agent in a natural manner, on reducing the burden on the instructor, and on focusing learning on the procedure being taught. Initially the agent has little domain knowledge. The instructor demonstrates a procedure by directly manipulating a simulated environment. However, one demonstration is not sufficient for understanding the causal relationships between a demonstration's actions. Unfortunately, the more demonstrations a procedure requires, the greater the instructor's burden. However, fewer demonstrations can be required if the agent autonomously experiments. Our...
Embodied conversational agents (ECA) have potential as facilitators for health interventions. How... more Embodied conversational agents (ECA) have potential as facilitators for health interventions. However, their utility is limited as long as people must sit down in front of a computer to access them. This paper describes a project that is deploying an ECA on a handheld computer, and using it to assist in a psychosocial intervention aimed at providing training in problem solving skills. The agent is based upon the virtual trainer/counselor in the pedagogical drama Carmen's Bright IDEAS, adapted for handheld use and for interaction with a human caregiver. The system will go into clinical trails in August of 2004. The paper discusses the design and technical issues involved in the transition from laptop computer to handheld device and from 3rd-person view to first-person interaction, and the plan for evaluation. The clinical trial is designed both to evaluate psychosocial outcomes and to assess user preferences in ECA interaction modalities over the course of multiple sessions of use.
Plan recognition techniques frequently make rigid assumptions about the student's plans, and inve... more Plan recognition techniques frequently make rigid assumptions about the student's plans, and invest substantial effort to infer unobservable properties of the student. The pedagogical benefits of plan recognition analysis are not always obvious. We claim that these difficulties can be overcome if greater attention is paid to the situational context of the student's activity and the pedagogical tasks which plan recognition is intended to support. This paper describes an approach to plan recognition called situated plan attribution that takes these factors into account. It devotes varying amounts of effort to the interpretation process, focusing the greatest effort on interpreting impasse points, i.e., points where the student encounters some difficulty completing the task. This approach has been implemented and evaluated in the context of the REACT tutor, a trainer for Operators of deep space communications stations.
We envisioned that the ability of pedagogical agents to interact with learners in a natural, huma... more We envisioned that the ability of pedagogical agents to interact with learners in a natural, human-like way would make learning easier, more engaging, and more motivating. This deeper engagement, in turn, would result in improved learning outcomes. At the time, prototype pedagogical agents capable of human-like interaction with learners were only just beginning to be developed. Figure 1 shows two such agents, which we cite as examples in the following discussion. Herman the Bug, on the left, observed the actions of learners interacting with the Design-a-Plant game and provided problem-solving advice, explanations, and hints (Lester, Converse, Stone, et al. 1997). STEVE (Soar Training Expert for Virtual Environments), on the right, inhabited the VET (Virtual Environments for Training) virtual environment and coached learners in the operation of a shipboard power plant (Rickel and Johnson 1999).
Uploads
Papers by Lewis Johnson