Sign Language Avatars: Animation and Comprehensibility
2011, Lecture Notes in Computer Science
https://doi.org/10.1007/978-3-642-23974-8_13Abstract
Many deaf people have significant reading problems. Written content, e.g. on internet pages, is therefore not fully accessible for them. Embodied agents have the potential to communicate in the native language of this cultural group: sign language. However, state-of-the-art systems have limited comprehensibility and standard evaluation methods are missing. In this paper, we present methods and discuss challenges for the creation and evaluation of a signing avatar. We extended the existing EMBR character animation system 1 with prerequisite functionality, created a gloss-based animation tool and developed a cyclic content creation workflow with the help of two deaf sign language experts. For evaluation, we introduce delta testing, a novel way of assessing comprehensibility by comparing avatars with human signers. While our system reached stateof-the-art comprehensibility in a short development time we argue that future research needs to focus on nonmanual aspects and prosody to reach the comprehensibility levels of human signers.
References (31)
- Elliott, R., Glauert, J.R.W., Kennaway, J.R., Marshall, I., Safar, E.: Linguistic modelling and language-processing technologies for avatar-based sign language pre- sentation. Univers. Access Inf. Soc. 6, 375-391 (January 2008)
- Filhol, M.: Zebedee: a lexical description model for sign language synthesis. Tech. Rep. 2009-08, LIMSI (2009)
- Hanke, T.: iLex -a tool for sign language lexicography and corpus analysis. In: Proceedings of the 3rd International Conference on Language Resources and Eval- uation. pp. 923-926 (2002)
- Heloir, A., Kipp, M.: Realtime animation of interactive agents: Specification and realization. Journal of Applied Artificial Intelligence 24(6), 510-529 (2010)
- Heloir, A., Kipp, M., Gibet, S., Courty, N.: Specifying and evaluating data-driven style transformation for gesturing embodied agents. In: Proceedings of the 8th In- ternational Conference on Intelligent Virtual Agents (IVA-08). pp. 215-222. LNAI 5208, Springer (2008)
- Hodgins, J., Jörg, S., O'Sullivan, C., Park, S.I., Mahler, M.: The saliency of anoma- lies in animated human characters. ACM Trans. Appl. Percept. 7, 22:1-22:14 (July 2010)
- Holt, J.A.: Demographic, Stanford achievement test -8th edition for deaf and hard of hearing students: Reading comprehension subgroup results. Amer. Annals Deaf 138, 172-175 (1993)
- Huenerfauth, M.: A linguistically motivated model for speed and pausing in anima- tions of american sign language. ACM Trans. Access. Comput. 2, 9:1-9:31 (June 2009)
- Huenerfauth, M., Hanson, V.L.: Sign language in the interface: Access for deaf signers. In: Stephanidis, C. (ed.) The Universal Access Handbook. CRC Press (2009)
- Huenerfauth, M., Zhao, L., Gu, E., Allbeck, J.: Evaluating american sign language generation through the participation of native ASL signers. In: Proc. of the 9th International ACM Conference on Computers and Accessibility (ASSETS). pp. 211-218. ACM, New York, NY, USA (2007)
- Huenerfauth, M., Zhao, L., Gu, E., Allbeck, J.: Evaluating american sign language generation by native ASL signers. ACM Transactions on Access Computing 1(1), 1-27 (May 2008)
- Johnston, T.: Australian Sign Language (Auslan): An introduction to sign language linguistics. Cambridge University Press (2007)
- Kennaway, J.R., Glauert, J.R.W., Zwitserlood, I.: Providing signed content on the internet by synthesized animation. ACM Transactions on Computer-Human Interaction (TOCHI) 14(3), 15-29 (2007)
- Kipp, M.: Anvil: The video annotation research tool. In: Durand, J., Gut, U., Kristofferson, G. (eds.) Handbook of Corpus Phonology. Oxford University Press (2011), to appear
- Kipp, M., Heloir, A., Gebhard, P., Schröder, M.: Realizing multimodal behavior: Closing the gap between behavior planning and embodied agent presentation. In: Proceedings of the 10th International Conference on Intelligent Virtual Agents (IVA-10). Springer (2010)
- Kipp, M., Neff, M., Albrecht, I.: An Annotation Scheme for Conversational Ges- tures: How to economically capture timing and form. Journal on Language Re- sources and Evaluation -Special Issue on Multimodal Corpora 41(3-4), 325-339 (December 2007)
- Kipp, M., Nguyen, Q.: Multitouch Puppetry: Creating coordinated 3D motion for an articulated arms. In: Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (2010)
- Kipp, M., Nguyen, Q., Heloir, A., Matthes, S.: Assessing the deaf user perspective on sign language avatars. In: Proceedings of the 13th International ACM SIGAC- CESS Conference on Computers and Accessibility (ASSETS). ACM Press (2011)
- Kita, S., van Gijn, I., van der Hulst, H.: Movement phases in signs and co-speech gestures, and their transcription by human coders. In: Wachsmuth, I., Fröhlich, M. (eds.) Gesture and Sign Language in Human-Computer Interaction. pp. 23-35. Springer, Berlin (1998)
- Liddell, S.K., Johnson, R.E.: American sign language: The phonological base. Sign Language Studies 64, 195-277 (1989)
- McNeill, D.: Hand and Mind: What Gestures Reveal about Thought. University of Chicago Press, Chicago (1992)
- Neff, M., Kipp, M., Albrecht, I., Seidel, H.P.: Gesture Modeling and Animation Based on a Probabilistic Recreation of Speaker Style. ACM Transactions on Graph- ics 27(1), 1-24 (March 2008)
- Niewiadomski, R., Bevacqua, E., Mancini, M., Pelachaud, C.: Greta: An inter- active expressive ECA systems. In: Proc. of the 8th International Conference on Autonomous Agents and Multiagent Systems (AAMAS). pp. 1399-1400 (2009)
- Pfau, R., Quer, J.: Nonmanuals: their prosodic and grammatical roles. In: Brentari, D. (ed.) Sign languages (Cambridge Language Surveys), pp. 381-402. Cambridge University Press (2010)
- Prillwitz, S., Leven, R., Zienert, H., Hanke, T., Henning, J.: HamNoSys Version 2.0. Hamburg Notation System for Sign Language. An Introductory Guides. Signum (1989)
- Schröder, M., Trouvain, J.: The german text-to-speech synthesis system mary: A tool for research, development and teaching. International Journal of Speech Technology 6, 365-377 (2003)
- Sheard, M., Schoot, S., Zwitserlood, I., Verlinden, M., Weber, I.: Evaluation reports 1 and 2 of the EU project essential sign language information on government networks, Deliverable D6.2 (March 2004)
- Stokoe, W.C.: Sign language structure: An outline of the visual communication system of the American deaf. Studies in linguistics, Occasional papers 8 (1960)
- Swisher, V., Christie, K., Miller, S.: The reception of signs in peripheral vision by deaf persons. Sign Language Studies 63, 99-125 (1989)
- Thiebaux, M., Marshall, A., Marsella, S., Kallman, M.: SmartBody: Behavior re- alization for embodied conversational agents. In: Proc. of the 7th Int. Conf. on Autonomous Agents and Multiagent Systems (AAMAS) (2008)
- Wolfe, R., McDonald, J., Davidson, M.J., Frank, C.: Using an animation-based technology to support reading curricula for deaf elementary schoolchildren. In: The 22nd Annual International Technology & Persons with Disabilities Conference (2007)