Embodied Conversational Agents

This article describes research to build an embodied conversational agent (ECA) as an interface to a question-and-answer (Q/A) system about a National Science Foundation (NSF) program. We call this ECA the LifeLike Avatar, and it can interact with its users in spoken natural language to answer general as well as specific questions about specific topics. In an idealized case, the LifeLike Avatar could conceivably provide a user with a level of interaction such that he or she would not be certain as to whether he or she is talking to the actual person via video teleconference. This could be considered a (vastly) extended version of the seminal Turing test. Although passing such a test is still far off, our work moves the science in that direction. The Uncanny Valley notwithstanding, applications of such lifelike interfaces could include those where specific instructors/caregivers could be represented as stand-ins for the actual person in situations where personal representation is important. Possible areas that come to mind that might benefit from these lifelike ECAs include health-care support for elderly/disabled patients in extended home care, education/training, and knowledge preservation. Another more personal application would be to posthumously preserve elements of the persona of a loved one by family members. We apply this approach to a Q/A system for knowledge preservation and dissemination, where the specific individual who had this knowledge was to retire from the US National Science Foundation. The system is described in detail, and evaluations were performed to determine how well the system was perceived by users.

Resumo: Este artigo irá abordar a utilização da Inteligência Artificial (IA) na Educação a Distância (EAD). A IA pode ser uma ferramenta poderosa para personalizar a experiência de aprendizagem dos alunos, fornecer feedback instantâneo sobre o desempenho, criar meios de assistências virtuais para tirar dúvidas e analisar dados para melhorar a eficácia dos cursos realizados a distância. Neste contexto, é importante destacar, a importância em considerar o uso da Inteligência artificial, em benefício da educação em geral, na contemporaneidade. No entanto, é preciso considerar e proteger a privacidade dos dados dos alunos e a ética no uso da IA. Além disso, a implementação da IA pode exigir custos

We present work towards a spoken-dialogue embodied companion system that behaves in a socially cooperative fashion, i.e., being attentive to communicative and social cues emitted by the user, inferring internal cognitive, affective and interactional states, and reciprocating with appropriately adapted behavior. It is argued that this capability is vital for fostering cooperative, effortless, and enjoyable interactions that maximize both efficacy and long-term acceptance with elderly or cognitively impaired users.

People of low literacy could benefit from automated support when learning about societal participation. We design an Embodied Conversational Agent (ECA) 'coach' that can provide effective learning support to low-literate learners, develop a prototype virtual learning environment, and evaluate this prototype with low-literate end users. First, we inventory the learning support benefits of ECA coaching. Second, we update existing requirements to better specify functional demands for the coach ECA. Third, we write use cases and develop the prototype. Finally, we evaluate the prototype with low-literate users in a mixed-method within-subjects experiment. Results show that the coach influences the subjective learning experience: Participants report higher positive affect, higher user-system engagement, and increased self-efficacy regarding online banking. These results particularly apply to the domain of challenging information skills exercises. Caveats apply: One of four exercises was significantly more difficult than the other three; and coach support rules were not clearly formalized.

Dans cet article, nous présentons nos travaux en cours autour des agents conversationnels oraux virtuels et de leur adaptation en ligne aux utilisateurs. En particulier plusieurs possibilités d'introduction de productions de traits humoristiques dans les systèmes de dialogue oral sont étudiées, afin de rendre plus naturelles les interactions entre un agent et les utilisateurs. Enfin, nous discutons comment les sciences cognitives, et en particulier les neurosciences, peuvent aider à évaluer de manière plus complète les systèmes et les évolutions proposés en les comparant, à la fois objectivement et subjectivement, aux situations d'interactions sociales entre humains.

People of low literacy could benefit from automated support when learning about societal participation. We design an Embodied Conversational Agent (ECA) 'coach' that can provide effective learning support to low-literate learners, develop a prototype virtual learning environment, and evaluate this prototype with low-literate end users. First, we inventory the learning support benefits of ECA coaching. Second, we update existing requirements to better specify functional demands for the coach ECA. Third, we write use cases and develop the prototype. Finally, we evaluate the prototype with low-literate users in a mixed-method within-subjects experiment. Results show that the coach influences the subjective learning experience: Participants report higher positive affect, higher user-system engagement, and increased self-efficacy regarding online banking. These results particularly apply to the domain of challenging information skills exercises. Caveats apply: One of four exercises was significantly more difficult than the other three; and coach support rules were not clearly formalized.

A evolução tecnológica tem contribuído significativamente para mudanças relevantes em quase todas as dimensões da nossa sociedade, nomeadamente na educação e, consequentemente, na Educação a Distância (EaD). Associando as potencialidades das tecnologias disponíveis, à necessidade dos estudantes e à indispensabilidade das instituições automatizarem alguns processos passíveis de serem automatizados, justifica-se a criação de um Tutor Virtual que execute algumas funções que contribuam para as situações de aprendizagem dos estudantes. Desta forma, a presente investigação teve como finalidade analisar a integração de um Tutor Virtual em algumas Unidades Curriculares dos Cursos da Universidade Aberta, nas áreas de Ciências Naturais, Ciências Computacionais e Ciências Sociais, como complemento de Tutores Humanos. A integração de um Tutor Virtual, baseado em Agentes Virtuais Encapsulados (AVE), nos ambientes de Gestão de Sistemas de Aprendizagem (LMS), assentes na integração de interfaces antropomórficas, cujas aparências e características são tão realistas e credíveis e sem piorar o seu desempenho, faz com que este tipo de agentes sejam interlocutores para complementar os tutores humanos em ambientes reais de aprendizagem. A metodologia foi um estudo de caso e os instrumentos de recolha de dados foram: documentação do projeto, inquéritos por questionário (aos alunos participantes) e entrevistas semiestruturada (aos gestores/professores envolvidos no projeto). Em relação aos resultados, é de referir que o uso dos agentes encapsulados motivou e facilitou o processo de comunicação entre os estudantes e o Tutor, para além disso, proporcionou um acesso mais rápido aos conteúdos das UC.

Embodied conversational agents (ECA's) have become ubiquitous in human-computer interaction applications. Implementing humanlike multimodal behavior in these agents is difficult, because so little is known about the alignment of facial expression, eye gaze, gesture, speech and dialogue act. The current study used the data from an extensive study of human face-to-face multimodal communication for the development of a multimodal ECA, and tested to what extent multimodal behavior influenced the human-computer interaction. Results from a persona assessment questionnaire showed the presence of facial expressions, gesture and intonation had a positive effect on five assessment scales. Eye tracking results showed facial expressions played a primarily pragmatic role, whereas intonation played a primarily semantic role. Gestures played a pragmatic or semantic role, dependent on their level of specificity. These findings shed light on multimodal behavior within and between human and digital dialogue partners.

Game-based learning has been gaining popularity as a pedagogical approach in higher education. This study aimed to investigate the impact and effectiveness of game-based learning in higher education, specifically focusing on the interactive game The Mdina Castle’s Treasure Quest for international adult learners’ acquisition of Maltese prepositions. This study employed an experimental group that received instruction about Maltese prepositions through the game-based learning platform using The Mdina Castle’s Treasure Quest and a control group that received traditional lecture-based instruction from the same teacher. The experimental group consisted of 32 international adult students who embarked on the quest to find the hidden medieval gold coin treasure in the sprawling grand halls, underground secret chambers, and defensive towers of Mdina Castle. On the other hand, the control group comprised 31 adult students who received conventional classroom instruction. The results showed that the game-based learning group outperformed the control group significantly in post-instruction assessment about Maltese prepositions. The game-based learning group also reported higher engagement, motivation, and positive attitudes toward learning prepositions. This study provides evidence supporting the benefits of interactive games for language learning in adult learners, emphasizing the value of immersive learning experiences and offering insights into effective game-based language learning in higher education.

In recent years, game-based learning has emerged as a promising pedagogical approach in higher education. The idea behind game-based learning is that students can acquire knowledge and skills more effectively through engaging and interactive activities that mimic the mechanics of games (Karagiorgas & Niemann, 2017). This approach has gained traction due to its potential to promote active learning, enhance student engagement, and foster deeper understanding (Ismaizam et al., 2022). One area where game-based learning has shown promise is in language acquisition, particularly for adult learners. This chapter explores the impact of a specific game, The Mdina Castle's Treasure Quest, on prepositional learning in adult students. The game provided a captivating and immersive environment for 32 international adult students to learn and practice Maltese prepositions. By integrating prepositions into the gameplay, the students navigated through the virtual castle, solved puzzles, and overcame challenges to discover the hidden medieval gold coin treasure. The primary objective of this study was to investigate the effectiveness of game-based learning in higher education, specifically in the context of prepositional learning. This study aimed to evaluate the effect of the game on student performance and engagement by comparing the learning results of the game-based education group with a control group that received conventional lecture-based instruction. Additionally, it aimed to explore students' attitudes and motivations toward game-based language learning. 2.1.1 Research Questions This chapter seeks to address the following research questions: What is the impact on adult students of learning Maltese prepositions by playing the game The Mdina Castle's Treasure Quest? How does game-based instruction compare to traditional lecture-based instruction in terms of prepositional learning outcomes? What are the students' perceptions, attitudes, and levels of engagement toward learning prepositions through game-based instruction?

1. INTRODUCTION The history of human evolution has been one of adaptation both to the environment and to ourselves as a species. As our brains grew more subtle, so d id the mechanisms by which we communicate with on ea nother. Our verbal and nonverbal behaviors are wonderfully ...

has to be able to carry out general chat conversations and domain specific discussions as well. The structure of the dialogue graph and the attributes of the graph nodes have to support the success of the conversation: user utterance understanding and responding, as well as repairing the “non-understanding ” situations. In the following sections we are briefly depicting the system and describe its dialogue knowledge base focusing on the different user interfaces we developed for its testing and building.

Virtual worlds may be inhabited by intelligent agents who interact by performing various simple and complex actions. If the agents are human-like (embodied), their actions may be generated from motion capture or procedural animation. In this thesis, we introduce the CaPAR interactive system which combines both these approaches to generate agent-size neutral representations of actions within a framework called Parameterized Action Representation (PAR). Just as a person may learn a new complex physical task by observing another person doing it, our system observes a single trial of a human performing some complex task that involves interaction with self or other objects in the environment and automatically generates semantically rich information about the action. This information can be used to generate similar constrained motions for agents of different sizes. Human movement is captured by electromagnetic sensors. By computing motion zerocrossings and geometric spatial proximities, we isolate significant events, abstract both spatial and visual constraints from an agent's action, and segment a given complex action into several simpler subactions. We analyze each independently and build individual PARs for them. Several PARs can be combined into one complex PAR representing the original activity. Within each motion segment, semantic and style information is extracted. The style information is used to generate the same constrained motion in other differently sized virtual agents by copying the end-effector velocity profile, by following a similar end-effector trajectory, or by scaling and mapping force interactions between the agent and an object. The semantic information is stored in a PAR. The extracted style and constraint information is stored in the corresponding agent and object models.

We need oxygen, especially if someone farts!" The goal here was to determine whether computer interfaces are capable of social influence via humor. Users interacted with a natural language capable virtual agent that told persuasive information, and they were given the option to use information from the dialogue in order to complete a problem-solving task. Individuals interacting with an ostensibly humorous virtual agent were influenced by it such that those who judged the agent unfunny were less likely to be persuaded and departed from the agent's suggestions. We discuss the implications of these results for HCI involving natural language systems and virtual agents.

Designing virtual personal assistants that are able to engage users in an interaction have been a challenge for HCI researchers for the past 20 years. In this work we investigated how a set of vocal characteristics known as "conversational style" could play role in engaging users in an interaction with a virtual agent. We also examined whether the similarity attraction principle influences how people orient towards agents with different styles. Results of a within subject experiment on 102 subjects revealed that users exhibited similarity attraction toward computer agents, and preferred the agent whose conversational style matched their own. The study results contribute to our understanding of how the design of intelligent agents' conversational style influences users' engagement and perceptions of the agent, compared to known human-to-human interaction.

Multimodal corpora prove useful at different stages of the development process of embodied conversational agents. Insights into human-human communicative behaviors can be drawn from such corpora. Rules for planning and generating such behavior in agents can be derived from this information. And even the evaluation of human-agent interactions can rely on corpus data from human-human communication. In this paper, we exemplify how corpora can be exploited at the different development steps, starting with the question of how corpora are annotated and on what level of granularity. The corpus data can be used either directly for imitating the human behavior recorded in the corpus or rules can be derived from the data which govern the behavior planning process. Corpora can even play a vital role in the evaluation of agent systems. Several studies are presented that make use of corpora for the evaluation task.

While virtual humans are proven tools for training, education and research, they are far from realizing their full potential. Advances are needed in individual capabilities, such as character animation and speech synthesis, but perhaps more importantly, fundamental questions remain as to how best to integrate these capabilities into a single framework that allows us to efficiently create characters that can engage users in meaningful and realistic social interactions. This integration requires in-depth, inter-disciplinary understanding few individuals, or even teams of individuals, possess. We help address this challenge by introducing the ICT Virtual Human Toolkit 1 , which offers a flexible framework for exploring a variety of different types of virtual human systems, from virtual listeners and question-answering characters to virtual role-players. We show that due to its modularity, the Toolkit allows researchers to mix and match provided capabilities with their own, lowering the barrier of entry to this multidisciplinary research challenge.

While virtual humans are proven tools for training, education and research, they are far from realizing their full potential. Advances are needed in individual capabilities, such as character animation and speech synthesis, but perhaps more importantly, fundamental questions remain as to how best to integrate these capabilities into a single framework that allows us to efficiently create characters that can engage users in meaningful and realistic social interactions. This integration requires in-depth, inter-disciplinary understanding few individuals, or even teams of individuals, possess. We help address this challenge by introducing the ICT Virtual Human Toolkit 1 , which offers a flexible framework for exploring a variety of different types of virtual human systems, from virtual listeners and question-answering characters to virtual role-players. We show that due to its modularity, the Toolkit allows researchers to mix and match provided capabilities with their own, lowering the barrier of entry to this multidisciplinary research challenge.

Currently, there are many applications like human-computer interactions in which speech technology plays an important role. In particular, embodied conversational character interfaces research has produced widely divergent results and it has tended to focus on the character’s dialog capabilities associated with reasoning. Conversely, the present work attempts to evaluate a conversational character from interactionrelated aspects. This paper describes an analysis of a conversational character spoken dialogue system using a discrete event simulator. The simulation model was implemented in the ARENA traditional simulator. Simulation results show that sometimes the response time from the speech conversational character can be too long to user, as well as environment noise is an important aspect of the system to be improved.

We studied the role of the competence of a user interface agent/assistant that helped users to learn and use a new text editor. Participants in the study made a set of prescribed changes to a document via the editor with the aid of one of four interface agents. Participants could ask questions out loud to the agent and the agent would respond using a synthesized voice; the agent would also make proactive suggestions. The agents varied in the quality of responses and suggestions made. One group of participants were provided with a help screen as well as the agent. We focused on assessing the relation between users' objective performance, interaction style, and subjective experience. Results revealed that the perceived utility of the agent was influenced by the types of errors made by the agent, while participants' subjective impressions of the agent related to the perceptions of its representation. In addition, allowing participants to choose their preferred assistance style(s) (agent vs. onlinehelp) improved objective performance. We correlate quantitative findings with qualitative interview data and discuss implications for the design and the implementation of systems with interface agents.

This paper surveys some works on storytelling with robots. It tries to list type of robot used, who is the user of the system built, what is the focus of the study, and what is the outcome of the study. Most of the users are children, either normal or those with disabilities. The focus of the study includes robots as learning companions/pets, robot programming, interaction design techniques, technology introduction and pedagogy (robots as learning materials), and robots as teaching assistance. Meanwhile, the outcomes are a prototype, learning environment, authoring environment, and pedagogy experience.

This article describes research to build an embodied conversational agent (ECA) as an interface to a question-and-answer (Q/A) system about a National Science Foundation (NSF) program. We call this ECA the LifeLike Avatar, and it can interact with its users in spoken natural language to answer general as well as specific questions about specific topics. In an idealized case, the LifeLike Avatar could conceivably provide a user with a level of interaction such that he or she would not be certain as to whether he or she is talking to the actual person via video teleconference. This could be considered a (vastly) extended version of the seminal Turing test. Although passing such a test is still far off, our work moves the science in that direction. The Uncanny Valley notwithstanding, applications of such lifelike interfaces could include those where specific instructors/caregivers could be represented as stand-ins for the actual person in situations where personal representation is important. Possible areas that come to mind that might benefit from these lifelike ECAs include health-care support for elderly/disabled patients in extended home care, education/training, and knowledge preservation. Another more personal application would be to posthumously preserve elements of the persona of a loved one by family members. We apply this approach to a Q/A system for knowledge preservation and dissemination, where the specific individual who had this knowledge was to retire from the US National Science Foundation. The system is described in detail, and evaluations were performed to determine how well the system was perceived by users.

How do you develop a game that is provably educational, that is engaging from pre-school to adult, and that independently targets different language ability and world experience/maturity? Our purpose is to teach students of German or English as a Second Language in the same way that a baby learns a First Language. Children acquire language automatically without overt teaching, without conjugating verbs, looking up dictionaries or taking complex classes in syntax and morphology. They acquire through being immersed in an environment in which they have to learn to communicate in the language in order to achieve a variety of social and personal goals. In this project we provide a mixed environment with real toys for the learner to play with, and a simulated environment and a simulated teacher/caregiver to interact with. Whilst young children may be happy building towers and bridges out of wooden blocks, older children and adults cannot be expected to find motivating goals from the same ...

The quality of scientific oral presentations is often poor, owing to a number of factors, including public speaking anxiety. We present DynamicDuo, a system that uses an automated, life-sized, animated agent to help inexperienced scientists deliver their presentations in front of an audience. The design of the system was informed by an analysis of TED talks given by pairs of human presenters to identify the most common dual-presentation formats and transition behaviors used. We explore the usability and acceptability of DynamicDuo in both controlled laboratory-based studies and real-world environments, and its ability to decrease public speaking anxiety and improve presentation quality. In a within-subjects study (N 12) comparing co-presenting with DynamicDuo against solo-presenting with conventional presentation software, we demonstrated that our system led to significant improvements in public speaking anxiety and speaking confidence for non-native English speakers. Judges who viewed videotapes of these presentations rated those with DynamicDuo significantly higher on speech quality and overall presentation quality for all presenters. We also explore the affordances of the virtual co-presenter through empirical evaluation of novel roles the agent can play in scientific presentations and novel ways it can interact with the speaker in front of the audience.

Software agents and assistants, together with their adequate visual representations, lead to so-called social user interfaces, incorporating natural language interaction, context awareness and anthropomorphic avatars. Today's challenge is to build a suitable visualization architecture for anthropomorphic conversational user interfaces, and to design believable and appropriate faceto-face interactions, including human attributes, such as emotions. An integrated approach to these tasks is presented.

Autonomous Digital Actors represent the next step in authoring movies with believable characters, in a way that will allow them to be trained for acting specific roles in a story, suggesting appropriate behaviors during their performance. This article presents an overview of the art of acting and directing and how these concepts were used to elaborate a Virtual Actor metaphor. Also, we present an agent architecture for describing and implementing the virtual actors' acting knowledge base.

virtual reality is an important application area for research on multimodal dialogues because it allows more humanlike interactions among synthetic agents and humans than typical desktop interfaces can. To explore the use of animated agents for task-oriented collaboration in virtual worlds, we have designed such an agent: Steve (Soar Training Expert for Virtual Environments). Steve is fully implemented and integrated with the other software components on which he relies (i.e., visual interface software, a simulator, and commercial speech synthesis and recognition products). We have tested Steve on a variety of naval operating procedures; he can teach students how to operate and maintain the gas turbine engines aboard naval ships, including both individual tasks and team tasks. Steve is not limited to this domain; he can provide instruction in a new domain when given only the appropriate declarative domain knowledge. Despite the growing number of animated agents that converse with human users (André 1999; Johnson, Rickel, and Lester 2000; this book), Steve is unique in having domain-independent capabilities to support task-oriented dialogues situated in threedimensional virtual worlds. This chapter focuses on Steve's ability to integrate verbal and nonverbal communication to collaborate with students. Section 2 discusses the important roles that nonverbal communication plays in taskoriented collaboration. Section 3 illustrates Steve's current capabilities through an example interaction with a student. Sections 4 and 5 briefly review our architecture for virtual worlds and Steve's architecture; details are available in earlier papers (Johnson et al. 1998; Rickel and Johnson 1999a). Finally, section 6 describes the methods that govern Steve's communicative behavior, and section 7 provides conclusions and directions for future work. 2 Roles for Nonverbal Communication While most of the previous research on task-oriented dialogues has focused on verbal interactions, an animated agent that cohabits the virtual world with students permits other types of interactions that play important roles in human task-oriented dialogues. These roles for nonverbal communication provide the primary motivation for using animated agents for task-oriented collaboration.

Collection of patient-reported outcomes (PROs) remotely and their usage in the clinical workflow provide an improvement on both patient’s quality of life and cancer care. However, adoption of collecting PROs into the clinical workflow is rare, and existing works still have a lot of issues providing a holistic approach. This paper offers enhancements in the process of collecting PROs by utilization of conversational systems that still provide quite a new but promising way to collect PROs remotely with spoken interaction. Our proposed system provides an interoperability with Fast Healthcare Interoperability Resources (FHIR) server by using a multimodal sensing network (MSN) prepared for the project PERSIST. We introduce components of multimodality while collecting PROs with the help of the mHealth App and Open Health Connect (OHC) platform. As a result, chatbots and 3D embodied conversational agents (ECA) were prepared to interact with the cancer patients in 5 different languages. The...

As a result of the convergence of different services delivered over the internet protocol, internet protocol television (IPTV) may be regarded as the one of the most widespread user interfaces accepted by a highly diverse user domain. Every generation, from children to the elderly, can use IPTV for recreation, as well as for gaining social contact and stimulating the mind. However, technological advances in digital platforms go hand in hand with the complexity of their user interfaces, and thus induce technological disinterest and technological exclusion. Therefore, interactivity and affective content presentations are, from the perspective of advanced user interfaces, two key factors in any application incorporating human-computer interaction (HCI). Furthermore, the perception and understanding of the information (meaning) conveyed is closely interlinked with visual cues and non-verbal elements that speakers generate throughout human-human dialogues. In this regard, co-verbal behavior provides information to the communicative act. It supports the speaker's communicative goal and allows for a variety of other information to be added to his/her messages, including (but not limited to) psychological states, attitudes, and personality. In the present paper, we address complexity and technological disinterest through the integration of natural, human-like multimodal output that incorporates a novel combined data-and rule-driven co-verbal behavior generator that is able to extract features from unannotated, general text. The core of the paper discusses the processes that model and synchronize non-verbal features with verbal features even when dealing with unknown context and/or limited contextual information. In addition, the proposed algorithm incorporates data-driven (speech prosody, repository of motor skills) and rule-based concepts (grammar, gesticon). The algorithm firstly classifies the communicative intent, then plans the coverbal cues and their form within the gesture unit, generates temporally synchronized co-verbal cues, and finally realizes them in the form of human-like co-verbal movements. In this way, the information can be represented in the form of both meaningfully and temporally synchronized co-verbal cues with accompanying synthesized speech, using communication channels to which people are most accustomed.

Social skills training (SST) has recently emerged as a typical application for emotional conversational agents (ECA). While a number of prototypes have targeted the general population, fewer have been used for psychiatric patients despite the widely recognised potential of ECAs technologies in the field of mental health. Social cognition impairment is however a widely shared symptom in psychiatric patients suffering from pathologies such as schizophrenia. Going further than SST, rehabilitation programmes involving role-play, but also problem solving have been successfully used by clinicians, drastically improving the quality of life of patients suffering from such disabilities. One of the challenges of these programmes is to ensure that the patients will be able to adapt their behavior when the situation varies, rather than training them with the appropriate behavior for a set of specific situations. In this paper, we describe a novel approach for the development of a serious game s...

Rapid growth of competition on the electronic market place, will generate the demand for new innovative communication styles with web users. In this paper, we develop an operational approach for the automated generation of hypermedia presentations. Unlike conventional hypermedia, we use a lifelike presentation agent which presents the generated material, and guides the user through a dynamically expanding navigation space. The approach relies on a model that combines behavior planning for lifelike characters with concepts from hypermedia authoring such as timeline structures and navigation graphs.

With the advent of web browsers that are able to execute programs embedded in web pages, the use of animated characters for the presentation of information over the web has become possible. A strong argument in favour of using such characters in a web interface is the fact that they make human-computer interaction more enjoyable and allow for the emulation of communication styles common in humanhuman dialogue. In this paper we discuss three ongoing DFKI projects on lifelike synthetic characters in the internet. While all agents rely on the same approach for automated script generation, we use different player technologies which will be discussed in the light of different applications.

In this contribution, we argue in favor of a shift from applications with single presentation agents towards flexible performances given by a team of characters as a new presentation style. We will illustrate our approach by means of two subsequent versions of a test-bed called the "Inhabited Market Place" (IMP1 and IMP2). In IMP1, the attribute "flexible" refers to the system's ability to adapt a presentation to the needs and preferences of a particular user. In IMP2, flexibility additionally refers to the user's option of actively participating in a computer-based performance and influencing the behavior of the involved characters at runtime. While a plan-based approach has proven appropriate in both versions to automatically control the behavior of the agents, IMP2 calls for highly reactive and distributed behavior planning.

Embodied conversational characters are autonomous, graphically embodied virtual creatures that live in a 2D or 3D virtual environment. They are able to interact intelligently with human users, other characters, and their digital environment. While for decades research has concentrated on geometric body modelling and the development of animation and rendering techniques for virtual characters, other qualities have now come in focus as well, including the provision of conversational skills as well as the simulation of believable behavior including affect and peculiarities induced by individual personality traits. As a consequence, the domain of virtual characters has become much more diverse and now encompasses a wide range of disciplines, from computer graphics and animation to AI and more recently also psychology, sociology as well as design and arts. The current paper discusses a number of design issues that arise when building an application with one or more embodied characters. By means of selected sample applications we also illustrate a yet ongoing development of animated presentation agents starting with TV-style information presenters to highly interactive multi-character scenarios in which information is conveyed to the user in the form of multi-party conversations.

Personification of interface agents has been speculated to have several advantages, such as a positive effect on agent credibility and on the perception of learning experience. However, important questions less often addressed so far are what effect personification has on more objective measures, such as comprehension and recall, and furthermore, under what circumstances this effect (if any) occurs. We performed an empirical study with adult participants to examine the effect of the Ppp Persona not only on subjective but also on objective measures. In addition, we tested it both with technical and non-technical domain information. The results of the study indicate that the data from the subjective measures support the so called persona effect for the technical information but not for non-technical information. With regard to the objective measures, however, neither a positive nor a negative effect could be found. Implications for software development are discussed.

Advances in conversational interfaces have facilitated the widespread use of chatbots in assisting humans in their daily lives such as in retail services, healthcare, and education. With the use of Natural Language Processing techniques (NLP), chatbots can leverage on storytelling strategies to engage children in meaningful conversations to narrate and share their stories. In this paper, we compared two existing conversational storytelling agents, Orsen and EREN, by analyzing the dialog logs to examine children's engagement when sharing their stories with these agents. Taking three dimensions-language production, flow maintenance, and affect; into consideration, results show that the ability of EREN to recognize emotions expressed in the child's input text can encourage the children to be more open to sharing their stories. Even though the agent cannot generate affect-rich responses, its ability to recognize the emotions made children feel thankful and appreciative of the agent for listening. CCS CONCEPTS • Human-centered computing → Collaborative and social computing.

The evocation and detection of the user's emotional state is becoming a crucial element in the aim for developing more effective interfaces between humans and computers, especially in applications such as games and elearning tools. In this paper, we describe a model that allows us to determine emotion in real time, based on Electromyography (EMG) and skin conductance. The developed emotion recognition component has been used in a joint research project with the University of Bielefeld [1] that intends to address affective gaming by adding real time emotion detection to a game scenario between a human user and a 3D humanoid agent called Max.

In this paper we propose a modular structure to define communication acts with verbal and nonverbal elements inspired in the SAIBA model. Our modular structure is a conceptual interpretation of the functional features of a multimodal interaction platform we have developed, with an ...

This paper presents the NECA approach to the generation of dialogues between Embodied Conversational Agents (ECAs). This approach consist of the automated constructtion of an abstract script for an entire dialogue (cast in terms of dialogue acts), which is incrementally enhanced by a series of modules and finally "performed" by means of text, speech and body language, by a cast of ECAs. The approach makes it possible to automatically produce a large variety of highly expressive dialogues, some of whose essential properties are under the control of a user. The paper discusses the advantages and disadvantages of NECA's approach to Fully Generated Scripted Dialogue (FGSD), and explains the main techniques used in the two demonstrators that were built. The paper can be read as a survey of issues and techniques in the construction of ECAs, focussing on the generation of behaviour (i.e., focussing on information presentation) rather than on interpretation.

Interface agents are becoming a new way for computers to communicate with humans. These agents have gained much focus recently since there is a growing interest for presentations over the Internet. The application domain of these agents is becoming wider, the quality and complexity of the existing systems is increasing fast. Our contribution to this research field concerns a new system enabling authors to easily enhance their already existing content with synthetic agents having believable behavior. It consists of a customizable 3D facial agent system and a powerful language to author presentations using interface agents, called MPML. This system provides both a versatile agent and an easy-to- use control over it.

Software agents and assistants, together with their adequate visual representations, lead to so-called social user interfaces, incorporating natural language interaction, context awareness and anthropomorphic avatars. Today's challenge is to build a suitable visualization architecture for anthropomorphic conversational user interfaces, and to design believable and appropriate faceto-face interactions, including human attributes, such as emotions. An integrated approach to these tasks is presented.

In human-to-human communication, speech signals carry rich emotional cues that are further emphasized by affect-expressive gestures. In this regard, automatic synthesis and animation of gestures accompanying affective verbal communication can help to create more naturalistic virtual agents in human-computer interaction systems. Speech-driven gesture synthesis can map emotional cues of the speech signal to affect-expressive gestures by modeling complex variability and timing relationships of speech and gesture. In this paper, we investigate the use of continuous affect attributes, which are activation, valence and dominance, for speech-driven affective synthesis and animation of arm gestures. To this effect, we present a statistical framework based on hidden semi-Markov models (HSMM), where states are gestures and observations are speech-prosody and continuous affect attributes. The proposed framework is evaluated considering four distinct HSMM structures which differ by their emission distributions. Evaluations are performed over the USC CreativeIT database in a speaker-independent setup. Among the four statistical structures, the conditional structure, which models observation distributions as prosody given affect, achieves the best performance under both objective and subjective evaluations.