Academia.eduAcademia.edu

Outline

Title
Abstract
Introduction
Background
Question-Answering Characters
Conclusions and Future Work
References
All Topics
Computer Science
Natural Language Processing

All Together Now

Profile image of David TraumDavid Traum

2013, Lecture Notes in Computer Science

https://doi.org/10.1007/978-3-642-40415-3_33
visibility

…

description

14 pages

link

1 file

Abstract

While virtual humans are proven tools for training, education and research, they are far from realizing their full potential. Advances are needed in individual capabilities, such as character animation and speech synthesis, but perhaps more importantly, fundamental questions remain as to how best to integrate these capabilities into a single framework that allows us to efficiently create characters that can engage users in meaningful and realistic social interactions. This integration requires in-depth, inter-disciplinary understanding few individuals, or even teams of individuals, possess. We help address this challenge by introducing the ICT Virtual Human Toolkit 1 , which offers a flexible framework for exploring a variety of different types of virtual human systems, from virtual listeners and question-answering characters to virtual role-players. We show that due to its modularity, the Toolkit allows researchers to mix and match provided capabilities with their own, lowering the barrier of entry to this multidisciplinary research challenge.

Related papers

An Avatar Prepares: To what extent can a software-driven animated character simulate a human actor
Jon Weinbren
View PDFchevron_right
A Platform for Building Mobile Virtual Humans
Stacy Marsella

Intelligent Virtual Agents, 2015

We describe an authoring framework for developing virtual humans on mobile applications. The framework abstracts many elements needed for virtual human generation and interaction, such as the rapid development of nonverbal behavior, lip syncing to speech, dialogue management, access to speech transcription services, and access to mobile sensors such as the microphone, gyroscope and location components.

View PDFchevron_right
Toward virtual humans
William Swartout

AI Magazine, 2006

View PDFchevron_right
Toward a new generation of virtual humans for interactive experiences
Jonathan Gratch

2002

Abstract Virtual humans-autonomous agents that support face-to-face interaction in a variety of roles-can enrich interactive virtual worlds. Toward that end, the Mission Rehearsal Exercise project involves an ambitious integration of core technologies centered on a common representation of task knowledge.

View PDFchevron_right
Virtual Humans for Learning
David Traum

AI Magazine, 2013

Virtual humans are computer‐generated characters designed to look and behave like real people. Studies have shown that virtual humans can mimic many of the social effects that one finds in human‐human interactions such as creating rapport, and people respond to virtual humans in ways that are similar to how they respond to real people. We believe that virtual humans represent a new metaphor for interacting with computers, one in which working with a computer becomes much like interacting with a person and this can bring social elements to the interaction that are not easily supported with conventional interfaces. We present two systems that embody these ideas. The first, the twins are virtual docents in the Museum of Science, Boston, designed to engage visitors and raise their awareness and knowledge of science. The second, Sim‐Coach, uses an empathetic virtual human to provide veterans and their families with information about PTSD and depression.

View PDFchevron_right
Virtual character performance from speech
Stacy Marsella

: Our method can synthesize a virtual character performance from only an audio signal and a transcription of its word content. The character will perform semantically appropriate facial expressions and body movements that include gestures, lip synchronization to speech, head movements, saccadic eye movements, blinks and so forth. Our method can be used in various applications, such as previsualization tools, conversational agents, NPCs in video games, and avatars for interactive applications.

View PDFchevron_right
Steve goes to Bosnia: Towards a new generation of virtual humans for interactive experiences
William Swartout

2001

Abstract Interactive virtual worlds provide a powerful medium for entertainment and experiential learning. Our goal is to enrich such virtual worlds with virtual humansautonomous agents that support face-to-face interaction with people in these environments in a variety of roles. While supporting face-to-face interaction in virtual worlds is a daunting task, this paper argues that the key building blocks are already in place.

View PDFchevron_right
Engaging in a Conversation with Synthetic Characters Along the Virtuality Continuum
Elisabeth Andre

Lecture Notes in Computer Science, 2005

During the last decade research groups as well as a number of commercial software developers have started to deploy embodied conversational characters in the user interface especially in those application areas where a close emulation of multimodal human-human communication is needed. Most of these characters have one thing in common: In order to enter the user's physical world, they need to be physical themselves. The paper focuses on challenges that arise when embedding synthetic conversational agents in the user's physical world. We will start from work on synthetic agents that populate virtual worlds and anthropomorphic robots that inhabit physical worlds and discuss how the two areas need to be combined in order to populate physical worlds with synthetic characters. Finally, we will report on so-called traversable interfaces that allow agents to cross the border from the physical space to the virtual space and vice versa.

View PDFchevron_right
An extensible platform for interactive, entertaining social experiences with an animatronic character
Sabrina Haskell

Proceedings of the 2005 ACM SIGCHI International Conference on Advances in computer entertainment technology - ACE '05, 2005

The fields of human-robot interaction and entertainment robotics are truly interdisciplinary, combining the best of computer science, psychology, and mechanical engineering. However, they are areas which have thus far been largely limited to technical specialists, excluding animators, writers, and other artists from the development of interactive content. We attempt to open these fields to non-technologists through an easily extensible platform for the rapid development of social interactions between humans and animatronic characters. Specifically, the Interbots Platform consists of 1) a servo-driven animatronic character named Quasi and 2) a suite of tools to author decisions, animate the character, and design complete, interactive, entertaining social experiences. We have created a graphical editor to author behaviors for the character, as well as extensions for Alias Maya and Macromedia Director, empowering artists already familiar with these applications to create animatronic-enabled content. The usability of the Interbots Platform was verified when interdisciplinary student teams of artists and computer scientists were able to create engaging entertainment experiences with Quasi in two-week development cycles.

View PDFchevron_right
Social capabilities for autonomous virtual characters
Catherine Pelachaud

2006

Abstract. In this paper we describe our work toward the creation of affective multimodal virtual characters endowed with communicative and other socially significant capabilities. While much work in modern game AI has focused on issues such as path-finding and squad-level AI, more highly-detailed behaviour for small groups of interacting game characters has been lacking in the literature.

View PDFchevron_right

References (45)

  1. Rossen B., Lok B., A crowdsourcing method to develop virtual human conversational agents, International Journal of HCS, pp. 301-319 (2012)
  2. Bickmore, T.W., Bukhari L., Pfeiffer L., Paasche-Orlow M., Shanahan C., Hospital Bud- dy: A Persistent Emotional Support Companion Agent for Hospital Patients, Springer Ber- lin/Heidelberg, pp. 492-495 (2012)
  3. D'Mello, S. K., & Graesser, A. C. AutoTutor and affective AutoTutor: Learning by talking with cognitively and emotionally intelligent computers that talk back. ACM Transactions on Interactive Intelligent Systems, Volume 2, Issue 4, Article 23 (2012)
  4. Lane H.C., Noren,D., Auerbach D., Birch M., Swartout W., Intelligent Tutoring Goes to the Museum in the Big City: A Pedagogical Agent for ISE, Artificial Intelligence in Edu- cation, pp. 155-162 (2011)
  5. Johnson, W.L., Valente, A., Tactical Language and Culture Training Systems: Using AI to Teach Foreign Languages and Cultures. pp. 72-83 (2009)
  6. Campbell J., Core, M., Artstein R., Armstrong L., Hartholt A., Wilson C., Georgila K., Morbini F., Haynes E., Gomboc D., Birch M., Bobrow J., Lane H., Gerten J., Leuski A., Traum D., Trimmer M., DiNinni R., Bosack M., Jones T., Clark R., Yates K. Developing INOTS to support interpersonal skills practice. In Proceedings of the Thirty-second Annu- al IEEE Aerospace Conference, pp. 1-14, (2011)
  7. Pocketsphinx: A Free, Real-Time Continuous Speech Recognition System for hand-Held Devices, vol. 1, pp. 185-188 (2006)
  8. Littlewort G., Whitehill J., Wu T., Fasel I., Frank M., Movellan J., and Bartlett M., The Computer Expression Recognition Toolbox (CERT). Proc. IEEE International Conference on Automatic Face and Gesture Recognition (2011)
  9. Bickmore, T.W., Schulman D., Shaw, G., DTask & LiteBody: Open Source, Standards- based Tools for Building Web-deployed Embodied Conversational Agents, Intelligent Vir- tual Agents, PP. 425-431 (2009)
  10. Stone M., Specifying Generation of Referring Expressions by Example. AAAI Spring Symposium on NLG in Spoken and Written Dialogue, pp. 133-140 (2003)
  11. I.Poggi, C.Pelachaud, F. de Rosis, V. Carofiglio, B. De Carolis, GRETA. A Believable Embodied Conversational Agent, Multimodal Intelligent Information Presentation, (2005)
  12. Van Welbergen, H., Reidsma, D., Ruttkay, Z.M., Zwiers, J.: Elckerlyc: A BML realizer for continuous, multimodal interaction with a virtual human. Multimodal UI (2010)
  13. Shapiro, A.: Building a Character Animation System. In: The Fourth International Confer- ence on Motion in Games, Edinburgh, UK, November, 2011
  14. Heloir, A. and Kipp, M.: A Realtime Engine for Interactive Embodied Agents, Intelligent Virtual Agents, pp. 393-404, 2009
  15. Taylor, P., Black, A., Caley, R., The architecture of the Festival speech synthesis system. Third ESCA Workshop in Speech Synthesis, pp. 147-151. (1998)
  16. Heylen, D.K.J., Kopp, S., Marsella, S.C., Pelachaud, C., Vilhjálmsson, H.H. The Next Step towards a Function Markup Language. IVA (2008)
  17. Kopp, S., Krenn, B., Marsella, S., Marshall, A., Pelachaud, C., Pirker, H., Th´orisson, K., Vilhj´almsson, H., Towards a Common Framework for Multimodal Generation: The Be- havior Markup Language (LNAI), vol. 4133, pp. 205-217. (2006)
  18. H. van Welbergen, Y. Xu, M. Thiebaux, W.W. Feng, J. Fu, D. Reidsma and A. Shapiro Demonstrating and Testing the BML Compliance of BML Realizers. IVA (2011)
  19. Schröder, M. (2010). The SEMAINE API: Towards a standards-based framework for building emotion-oriented systems. Advances in Human-Machine Interaction (2010)
  20. Quigley, M., Conley, K., Gerkey, B.P., Faust, J., Foote, T., Leibs, J., Wheeler, R., and Ng, A.Y., ROS: an open-source Robot Operating System, ICRA Open Source Software, (2009)
  21. Leuski, A and Traum, D. NPCEditor: Creating virtual human dialogue using information retrieval techniques. AI Magazine, 32(2):42-56, (2011).
  22. Leuski, A., Traum D.. A statistical approach for text processing in virtual humans. In Pro- ceedings of the 26th Army Science Conference, Orlando, Florida, USA, December 2008.
  23. Lee, J. and S. Marsella. Nonverbal Behavior Generator for Embodied Conversational Agents, IVA (2006)
  24. Feng, A.W., Xu, Y, Shapiro, A. : An Example-Based Motion Synthesis Technique for Lo- comotion and Object Manipulation, SIGGRAPH (2012)
  25. Thiebaux, M., Lance, B., Marsella, S.: Real-time Expressive Gaze for Virtual Humans, AAMAS, vol. 1, pp. 321-328 (2008)
  26. Kallmann, M., Marsella, S.: Hierarchical Motion Controllers for Real-time Autonomous Virtual Humans, Intelligent Virtual Agents, pp. 253-265, (2005)
  27. Feng, A.W., Huang, Y, Shapiro, A., An Analysis of Motion Blending Techniques, The Fifth International Conference on Motion in Games, Rennes, France, November 2012
  28. Feng, A.W., Huang, Y, Xu, Y, Shapiro, A., Automating the Transfer of a Generic Set of Behaviors Onto a Virtual Character, Conference on Motion in Games, (2012)
  29. J. Wagner, F. Lingenfelser, N. Bee, and E. Andre. Social signal interpretation (ssi). KI - Kuenstliche Intelligenz, 25:251-256, (2011)
  30. T. Baltrusaitis, P. Robinson, and L.-P. Morency. 3D constrained local model for rigid and non-rigid facial tracking. IEEE Computer Vision and Pattern Recognition, June (2012)
  31. L.-P. Morency, J. Whitehill, and J. Movellan. Generalized adaptive view-based appearance model: Integrated framework for monocular head pose estimation. Automatic Face and Gesture Recognition, pp. 1-8, (2008)
  32. E. Suma, B. Lange, A. Rizzo, D. Krum, and M. Bolas, "FAAST: The Flexible Action and Articulated Skeleton Toolkit," Proceedings of IEEE Virtual Reality, pp. 247-248, (2011).
  33. Scherer, S., Marsella, S., Stratou G., Xu X., Morbini F., Egan A., Rizzo A., Morency, L.P., Perception Markup Language: Towards a Standardized Representation of Perceived Non- verbal Behaviors, IVA pp. 455-463 (2012)
  34. Leuski A, Kennedy B, Patel R, Traum DR. Asking questions to limited domain virtual characters: how good does speech recognition have to be? ASC (2006)
  35. Artstein R, Gandhe S, Leuski A, Traum DR. Field Testing of an interactive question- answering character. ELRA, LREC (2008)
  36. Swartout W, Traum DR, Artstein R, Noren D, Debevec P, Bronnenkant K, et al. Ada and Grace: Toward Realistic and Engaging Virtual Museum Guides. IVA pp. 286-300 (2010)
  37. Jonathan Gratch, Anna Okhmatovskaia, Francois Lamothe, Stacy Marsella, Mathieu Mo- rales, R. J. van der Werf, Louis-Philippe Morency, Virtual Rapport, IVA (2006)
  38. Rizzo A., Forbell E., Lange B., Buckwalter J.G., Williams J., Sagae K., Traum D., SimCoach: An Online Intelligent Virtual Agent System for Breaking Down Barriers to Care for Service Members and Veterans, Chapter in Healing War Trauma (2012) 45.
  39. Morbini F., DeVault D., Sagae K., Gerten J., Nazarian A., Traum D., FLoReS: A Forward Looking, Reward Seeking, Dialogue Manager, Spoken Dialog Systems (2012)
  40. Hartholt A., Gratch J., Weiss L., Leuski A., Morency L.P., Marsella S, At the Virtual Fron- tier: Introducing Gunslinger, a Multi-Character, Mixed-Reality, Story-Driven Experience IVA pp. 500-501 (2009)
  41. Kenny, P.G., Parsons, T.D., Gratch, J. Rizzo, A., Evaluation of Justina: A Virtual Patient with PTSD, Lecture Notes in Computer Science, pp. 394-408 (2008)
  42. Traum DR, Marsella S, Gratch J, Lee J, Hartholt A. Multi-party, Multi-issue, Multi- strategy Negotiation for Multi-modal Virtual Agents. IVA pp. 117-130(2008)
  43. Gratch J., Hartholt A., Dehghani M., Marsella S., Virtual Humans: A New Toolkit for Cognitive Science Research, CogSci (2013)
  44. Khooshabeh P., McCall C., Gandhe S., Gratch J., Blascovich J., Does it matter if a com- puter jokes, Extended abstracts on Human Factors in Computer Systems, pp. 77-86 (2011)
  45. Batrinca, L., Stratou, G., Morency, L.P., Scherer, S. Cicero -Towards a Multimodal Virtu- al Audience Platform for Public Speaking Training. IVA (2013)

Related papers

All Together Now: Introducing the Virtual Human Toolkit
David Traum

Intelligent Virtual Agents, 2013

While virtual humans are proven tools for training, education and research, they are far from realizing their full potential. Advances are needed in individual capabilities, such as character animation and speech synthesis, but perhaps more importantly, fundamental questions remain as to how best to integrate these capabilities into a single framework that allows us to efficiently create characters that can engage users in meaningful and realistic social interactions. This integration requires in-depth, inter-disciplinary understanding few individuals, or even teams of individuals, possess. We help address this challenge by introducing the ICT Virtual Human Toolkit 1 , which offers a flexible framework for exploring a variety of different types of virtual human systems, from virtual listeners and question-answering characters to virtual role-players. We show that due to its modularity, the Toolkit allows researchers to mix and match provided capabilities with their own, lowering the barrier of entry to this multidisciplinary research challenge.

View PDFchevron_right
Creating interactive virtual humans: Some assembly required
Elisabeth André

Intelligent Systems, …, 2002

View PDFchevron_right
Building Smart Embodied Virtual Characters
Elisabeth André

Lecture Notes in Computer Science, 2003

Embodied conversational characters are autonomous, graphically embodied virtual creatures that live in a 2D or 3D virtual environment. They are able to interact intelligently with human users, other characters, and their digital environment. While for decades research has concentrated on geometric body modelling and the development of animation and rendering techniques for virtual characters, other qualities have now come in focus as well, including the provision of conversational skills as well as the simulation of believable behavior including affect and peculiarities induced by individual personality traits. As a consequence, the domain of virtual characters has become much more diverse and now encompasses a wide range of disciplines, from computer graphics and animation to AI and more recently also psychology, sociology as well as design and arts. The current paper discusses a number of design issues that arise when building an application with one or more embodied characters. By means of selected sample applications we also illustrate a yet ongoing development of animated presentation agents starting with TV-style information presenters to highly interactive multi-character scenarios in which information is conveyed to the user in the form of multi-party conversations.

View PDFchevron_right
Using Virtual Humans to Bootstrap the Creation of Other Virtual Humans
Juan Cendan

Lecture Notes in Computer Science, 2010

Virtual human (VH) experiences are increasingly used for training interpersonal skills such as military leadership, classroom education, and doctor-patient interviews. These diverse applications of conversational VHs have a common and unexplored thread -a significant additional population would be afforded interpersonal skills training if VHs were available to simulate either interaction partner. We propose a computer-assisted approach to generate a virtual medical student from hundreds of interactions between a virtual patient and real medical students. This virtual medical student is then used to train standardized patients -human actors who roleplay the part of patients in practice doctor-patient encounters. Practice with a virtual medical student is expected to lead to greater standardization of roleplay encounters, and more accurate evaluation of medical student competency. We discuss the method for generating VHs from an existing corpus of human-VH interactions and present observations from a pilot experiment to determine the utility of the virtual medical student for training.

View PDFchevron_right
Imparting individuality to virtual humans
Arjan Egges, Nadia Magnenat-halmann

2002

In this paper, we present an integrated method of linking personality and emotion with the response generation and expression synthesis of Virtual Humans. The characters are powered by a dialogue system that consists of a large set of basic interactions between user and computer. These interactions are encoded in finite state machines. Transitions are linked with conditions and actions that can be connected with external modules. One of these modules is a personality module. In this way, responses of the virtual human depend not only on input given by a user, but also on its personality and emotional state. The dialogue system is connected to a 3D face that performs the speech and facial animation, together with facial expressions that reflect the personality specification.

View PDFchevron_right

Related topics

  • Computer Science
  • Human Computer Interaction
  • Natural Language Processing
  • Animation
  • Nonverbal Behavior
  • Medicine
  • Embodied Conversational Agents
  • Virtual Actor

    • Cited by

    Modeling Warmth and Competence in Virtual Characters
    Nhi Ngo, Truong-Huy Nguyen, Elín Carstensdóttir

    Lecture Notes in Computer Science, 2015

    Developing believable virtual characters has been a subject of research in many fields including graphics, animations, artificial intelligence, and humancomputer interaction. One challenge towards commoditizing the use of virtual humans is the ability to algorithmically construct characters of different stereotypes. In this paper, we present our efforts in designing virtual characters that can exhibit non-verbal behaviors to reflect varying degrees of warmth and competence, two personality traits shown to underlie social judgments and form stereotypical perception. To embark on developing a computational behavior model that portrays these traits, we adopt an iterative design methodology tuning the design using theory from theatre, animation and psychology, expert reviews, user testing and feedback. Using this process we were able to construct a set of virtual characters that portray variations of warmth and competence through combination of gestures, use of space, and gaze behaviors. In this paper we discuss the design methodology, the resultant system, and initial experiment results showing the promise of the model.

    View PDFchevron_right
    Interviewing Suspects with Avatars: Avatars Are More Effective When Perceived as Human
    Ellen Giebels

    Frontiers in psychology, 2016

    It has been consistently demonstrated that deceivers generally can be discriminated from truth tellers by monitoring an increase in their physiological response. But is this still the case when deceivers interact with a virtual avatar? The present research investigated whether the mere "belief" that the virtual avatar is computer or human operated forms a crucial factor for eliciting physiological cues to deception. Participants were interviewed about a transgression they had been seduced to commit, by a human-like virtual avatar. In a between-subject design, participants either deceived or told the truth about this transgression. During the interviews, we measured the physiological responses assessing participants' electrodermal activity (EDA). In line with our hypothesis, EDA differences between deceivers and truth tellers only were significant for participants who believed they interacted with a human operated (compared to a computer operated) avatar. These results ...

    View PDFchevron_right
    Flipper 2.0
    Jelte van Waterschoot

    Proceedings of the 18th International Conference on Intelligent Virtual Agents - IVA '18, 2018

    We present a new dialogue engine called Flipper 2.0 (Flipper) which aims to help developers of embodied conversational agents (ECAs) to quickly and flexibly create dialogues. Flipper provides a technically stable and robust dialogue management system to integrate with other components of ECAs such as behaviour realisers. We compare Flipper with state-of-the-art dialogue design systems. We describe the details of our dialogue engine, how it handles dialogue management and how it supports the authoring of dialogues. We demonstrate the use of the dialogue engine with examples of design patterns and discuss practical applications. Finally we give recommendations on the cases in which it is beneficial to use Flipper. CCS CONCEPTS • Human-centered computing → Natural language interfaces; Systems and tools for interaction design; User interface toolkits; • Information systems → Open source software;

    View PDFchevron_right
    The UAN Colombian co-speech gesture corpus
    Douglas Niño

    Lang. Resour. Evaluation, 2021

    A research group from the Universidad Antonio Narino at Bogota, Colombia, collected over four hours of audiovisual interaction between the Dean of the Faculty of Systems Engineering and prospective Colombian students to help answer questions about how to annotate Colombian gesture during spontaneous speech. Out of 10 interviews conducted during two semesters showing dyadic or multiparty interactions and recorded from three angles, the group selected five dyadic interactions from the second semester totaling over an hour and a half to code for gesture and speech behaviors. The group transcribed the speech and coded the behaviors of the Dean using the NEUROGES-ELAN coding system and the ELAN annotation software, adjusted to the group’s purposes. The group was able to achieve acceptable inter-rater reliability (around 80%) for all NEUROGES-ELAN modules. From the last Function category tier in NEUROGES-ELAN, which corresponds to conversational gestures, the group further determined lexi...

    View PDFchevron_right
    What Kind of Stories Should a Virtual Human Swap?
    Setareh Nasihati Gilani

    Intelligent Virtual Agents, 2016

    Telling stories is an important aspect virtual agents designed to interact with people socially over time. We describe an experiment designed to investigate the impact of the identity, presentation form, and perspective of a virtual storyteller on a human user who engages in a story-swapping activity with two virtual characters. For each interaction, the user was given 10 "ice-breaker" questions to ask a virtual character and respond to the character's reciprocal request. Participants also filled out a post-interaction survey, measuring rapport with the character and impressions of the character's personality. Results generally show that participants prefer characters who tell first person stories, however there were some interactions with presentation order. No significant preferences were established for the form or identity variables.

    View PDFchevron_right
    580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved