Academia.eduAcademia.edu

Outline

Title
Abstract
Introduction
Scope of This Work
Related Work
Method
Conclusion Stage
Results and Discussion
Conclusion
References
All Topics
Computer Science
Media Technology

Crowdsourcing authoring of sensory effects on videos

Profile image of Marcello de AmorimMarcello de Amorim

2019, Multimedia Tools and Applications

https://doi.org/10.1007/S11042-019-7312-2
visibility

…

description

27 pages

link

1 file

Abstract

Human perception is inherently multi-sensorial involving five traditional senses: sight, hearing, touch, taste, and smell. In contrast to traditional multimedia, based on audio and visual stimuli, mulsemedia seek to stimulate all the human senses. One way to produce multisensorial content is authoring videos with sensory effects. These effects are represented as metadata attached to the video content, which are processed and rendered through physical devices into the user's environment. However, creating sensory effects metadata is not a trivial activity because authors have to identify carefully different details in a scene such as the exact point where each effect starts, finishes, and also its presentation features such as intensity, direction, etc. It is a subjective task that requires accurate human perception and time. In this article, we aim at finding out whether a crowdsourcing approach is suitable for authoring coherent sensory effects associated with video content. Our belief is that the combination of a collective common sense to indicate time intervals of sensory effects with an expert fine-tuning is a viable way to generate sensory effects from the point of view of users. To carry out the experiment, we selected three videos from a public mulsemedia dataset, sent them to the crowd through a cascading microtask approach. The results showed that the crowd can indicate intervals in which users agree that there should be insertions of sensory effects, revealing a way of sharing authoring between the author and the crowd.

Related papers

Sensory Experience for Videos on the Web
Benjamin Rainer

2011 Workshop on Multimedia on the Web, 2011

More and more multimedia content is becoming available via the World Wide Web (WWW). These contents stimulate only senses like hearing or vision. Recently, it has been proposed to stimulate also other senses while consuming multimedia content, through so-called sensory effects. These sensory effects aim at enhancing the user's viewing experience by providing effects such as light, wind, vibration, etc. These effects are represented as Sensory Effect Metadata (SEM) which is associated to multimedia content and is rendered on devices like fans, lamps, or vibration chairs. In this paper, we present a plugin for Web browsers which is able to render such sensory effects provided via Web content, and we describe a demonstrator that uses this plug-in to control an amBX system.

View PDFchevron_right
A collaborative approach for semantic time-based video annotation using gamification
Paula Viana

Human-centric Computing and Information Sciences

Efficient access to large scale video assets, may it be our life memories in our hard drive or a broadcaster archive which the company is eager to sell, requires content to be conveniently annotated. Manually annotating video content is, however, an intellectually expensive and time-consuming process. In this paper we argue that crowdsourcing, an approach that relies on a remote task force to perform activities that are costly or time-consuming using traditional methods, is a suitable alternative and we describe a solution based on gamification mechanisms for collaboratively collecting timed metadata. Tags introduced by registered players are validated based on a collaborative scoring mechanism that excludes erratic annotations. Voting mechanisms, enabling users to approve or refuse existing tags, provide an extra guarantee on the quality of the annotations. The sense of community is also created as users may watch the crowd’s favourite moments of the video provided by a summarizati...

View PDFchevron_right
Using a new video rating tool to crowd-source analysis of behavioural reaction to stimuli
Holly Root-Gutteridge

Animal Cognition

Quantifying the intensity of animals’ reaction to stimuli is notoriously difficult as classic unidimensional measures of responses such as latency or duration of looking can fail to capture the overall strength of behavioural responses. More holistic rating can be useful but have the inherent risks of subjective bias and lack of repeatability. Here, we explored whether crowdsourcing could be used to efficiently and reliably overcome these potential flaws. A total of 396 participants watched online videos of dogs reacting to auditory stimuli and provided 23,248 ratings of the strength of the dogs’ responses from zero (default) to 100 using an online survey form. We found that raters achieved very high inter-rater reliability across multiple datasets (although their responses were affected by their sex, age, and attitude towards animals) and that as few as 10 raters could be used to achieve a reliable result. A linear mixed model applied to PCA components of behaviours discovered that...

View PDFchevron_right
Project CAVIAR CApturing VIewers’ Affective Response
Juan M Montero

Procesamiento Del Lenguaje Natural, 2019

In this project we propose the automatic analysis of the relation between the audiovisual characteristics of a multimedia production and the impact caused in its audience. With this aim, potential synergies are explored between different areas of knowledge including, among others: audiovisual communication, computer vision, multimodal systems, biometric sensors, social network analysis, opinion mining, and affective computing. Our efforts are oriented towards combining these technologies to introduce novel computational models that could predict the reactions of spectators to multimedia elements across different media and moments. On the one hand, we study the cognitive and emotional response of the spectators while they are watching the media instances, using neuroscience techniques and biometric sensors. On the other hand, we also study the reaction shown by the audience on social networks by relying on the automatic collection and analysis of different metadata related to the media elements, such as popularity, sharing patterns, ratings and commentaries.

View PDFchevron_right
ThumbReels: query sensitive web video previews based on temporal, crowdsourced, semantic tagging
Myles Kilgallon Scott

During online search, the user's expectations often differ from those of the author. This is known as the "intention gap" and is particularly problematic when searching for and discriminating between online video content. An author uses description and meta-data tags to label their content, but often cannot predict alternate interpretations or appropriations of their work. To address this intention gap, we present ThumbReels, a concept for query-sensitive video previews generated from crowdsourced, temporally defined semantic tagging. Further, we supply an open-source tool that supports on-the-fly temporal tagging of videos, whose output can be used for later search queries. A first user study validates the tool and concept. We then present a second study that shows participants found ThumbReels to better represent search terms than contemporary preview techniques.

View PDFchevron_right
6 Seconds of Sound and Vision: Creativity in Micro-videos
Michele Trevisiol

IEEE Conference on Computer Vision and Pattern Recognition (CVPR'14), 2014

The notion of creativity, as opposed to related concepts such as beauty or interestingness, has not been studied from the perspective of automatic analysis of multimedia content. Meanwhile, short online videos shared on social media platforms, or micro-videos, have arisen as a new medium for creative expression. In this paper we study creative micro- videos in an effort to understand the features that make a video creative, and to address the problem of automatic detection of creative content. Defining creative videos as those that are novel and have aesthetic value, we conduct a crowd- sourcing experiment to create a dataset of over 3,800 micro-videos labelled as creative and non-creative. We propose a set of computational features that we map to the components of our definition of creativity, and conduct an analysis to determine which of these features correlate most with creative video. Finally, we evaluate a supervised approach to automatically detect creative video, with promising results, showing that it is necessary to model both aesthetic value and novelty to achieve optimal classification accuracy.

View PDFchevron_right
SceneNet : Crowd Sourcing of Audio Visual Information aiming to create 4 D video streams
Gabriel-Miro Muntean

2016

Message from MMTC Chair Dear MMTC colleagues: It is time for a change! Our signature E-Letter and R-Letter have been well recognized by our research community over the year. Given the popularity they have gained, we have been advised by IEEE ComSoc to brand them for better exposure. After a long period of consultation and discussion, we have been approved by IEEE ComSoc to brand both letters into one title. From March 2016, we will name the MMTC E-Letter as MMTC Communications-Frontiers, the MMTC R-Letter as MMTC Communications-Reviews. Again, I would like to thank the directors and editors for both letters, for their passionate services that have made our MMTC signature publications one of the greatest successes.

View PDFchevron_right
Assessing the quality of sensory experience for multimedia presentations
Benjamin Rainer

Signal Processing: Image Communication, 2012

View PDFchevron_right
A crowdsourcing framework for the production and use of film and television data
Carlos Eugenio Ovalle

New Review of Hypermedia and Multimedia, 2011

This paper outlines a framework that would enable the detailed indexing of film and television media through crowdsourcing. By making it easier to generate detailed data about these media on a large scale, fans and scholars can more efficiently produce a wide range of artifacts that reflect their interests in this content. Our development of a test collection included detailed indexing of 12 feature films and 8 television programs. We describe the conditions that make crowdsourcing an ideal approach for accomplishing this work on a larger scale; present a three-level development framework; and discuss how automated indexing, crowdsourcing quality, and copyright concerns might influence continued development of the project. Our framework highlights the potential of both multimedia indexing and crowdsourcing and can serve as a model for others embarking on projects that involve indexing, annotating, or labeling large multimedia collections.

View PDFchevron_right
Combining content-based analysis and crowdsourcing to improve user interaction with zoomable video
Axel Carlier

Proceedings of the 19th ACM international conference on Multimedia - MM '11, 2011

This paper introduces a new paradigm for interacting with zoomable video. Our interaction technique reduces the number of zooms and pans required by providing recommended viewports to the users, and replaces multiple zoom and pan actions with a simple click on the recommended viewport. The usefulness of our technique is visible in the quality of the recommended viewport, which needs to match the user intention, track movement in the scene, and properly frame the scene in the video. To this end, we propose a hybrid method where content analysis is complimented by the implicit feedback of a community of users in order to recommend viewports. We first compute preliminary sets of recommended viewports by analyzing the content of the video. These viewports allow tracking of moving objects in the scene, and are framed without violating basic aesthetic rules. To improve the relevance of the recommended viewports, we collect viewing statistics as users view a video, and use the viewports they select to reinforce the importance of certain recommendations and penalize others. New recommendations that are not previously recognized by content analysis may also emerge. The resulting recommended viewports converge towards the regions in the video that are relevant to users. A user study involving 70 participants shows that an user interface incorporating with our paradigm leads to more number of zooms, into more informative regions with fewer interactions.

View PDFchevron_right

References (46)

  1. Ademoye OA, Murray N, Muntean GM, Ghinea G (2016) Audio masking effect on inter-component skews in olfaction-enhanced multimedia presentations. ACM Trans Multimedia Comput Commun Appl 12(4):51:1-51:14. https://doi.org/10.1145/2957753
  2. Amorim MN, Neto FRA, Santos CAS (2018) Achieving complex media annotation through collective wisdom and effort from the crowd. In: 2018 25th international conference on systems, signals and image processing (IWSSIP). IEEE, pp 1-5. https://doi.org/10.1109/IWSSIP.2018.8439402
  3. Ballan L, Bertini M, Del Bimbo A, Seidenari L, Serra G (2011) Event detection and recognition for semantic annotation of video. Multimedia Tools Appl 51(1):279-302. https://doi.org/10.1007/s 11042-010-0643-7
  4. Bartocci S, Betti S, Marcone G, Tabacchiera M, Zanuccoli F, Chiari A (2015) A novel multimedia- multisensorial 4d platform. In: AEIT International annual conference (AEIT), 2015. IEEE, pp 1-6. https://doi.org/10.1109/AEIT.2015.7415215
  5. Chen J, Yao T, Chao H (2018) See and chat: automatically generating viewer-level comments on images. MTAP: Multimedia Tools Appl, 1-14. https://doi.org/10.1007/s11042-018-5746-6
  6. Cho H (2010) Event-based control of 4d effects using mpeg rose. Master's thesis, School of Mechanical, Aerospace and Systems Engineering, Division of Mechanical Engineering. Korea Advanced Institute of Science and Technology. Master's Thesis
  7. Choi B, Lee ES, Yoon K (2011) Streaming media with sensory effect. In: 2011 international con- ference on information science and applications (ICISA). IEEE, pp 1-6. https://doi.org/10.1109/IC ISA.2011.5772390
  8. Chowdhury SN, Tandon N, Weikum G (2016) Know2look: commonsense knowledge for visual search. In: Proceedings of the 5th workshop on automated knowledge base construction, pp 57-62
  9. Covaci A, Zou L, Tal I, Muntean GM, Ghinea G (2018) Is multimedia multisensorial?-a review of mulsemedia systems. ACM Comput Survey (CSUR) 51(5):91
  10. Cross A, Bayyapunedi M, Ravindran D, Cutrell E, Thies W (2014) Vidwiki: enabling the crowd to improve the legibility of online educational videos. In: Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. ACM, pp 1167-1175
  11. Di Salvo R, Spampinato C, Giordano D (2016) Generating reliable video annotations by exploit- ing the crowd. In: IEEE Winter conf. on applications of computer vision (WACV). IEEE, pp 1-8. https://doi.org/10.1109/WACV.2016.7477718
  12. Dumitrache A, Aroyo L, Welty C, Sips RJ, Levas A (2013) A.: "dr. detective": combining gamification techniques and crowdsourcing to create a gold standard in medical text. 16-31
  13. Egan D, Brennan S, Barrett J, Qiao Y, Timmerer C, Murray N (2016) An evaluation of heart rate and electrodermal activity as an objective qoe evaluation method for immersive virtual real- ity environments. In: 8th international conference on quality of multimedia experience (qoMEX'16). https://doi.org/10.1109/QoMEX.2016.7498964
  14. Foncubierta Rodríguez A, Müller H (2012) Ground truth generation in medical imaging: a crowdsourcing-based iterative approach. In: Proceedings of the ACM multimedia 2012 workshop on crowdsourcing for multimedia, CrowdMM '12. ACM, New York, pp 9-14. https://doi.org/10.1145 /2390803.2390808
  15. Galton F (1907) Vox populi (the wisdom of crowds). Nature 75(7):450-451
  16. Ghinea G, Timmerer C, Lin W, Gulliver SR (2014) Mulsemedia: State of the art, per- spectives, and challenges. ACM Trans Multimedia Comput Commun Appl 11(1s):17:1-17:23. https://doi.org/10.1145/2617994
  17. Gottlieb L, Choi J, Kelm P, Sikora T, Friedland G (2012) Pushing the limits of mechanical turk: qual- ifying the crowd for video geo-location. In: Proceedings of the ACM multimedia 2012 workshop on crowdsourcing for multimedia. ACM, pp 23-28
  18. Hardman L, Obrenović Ž, Nack F, Kerhervé B, Piersol K (2008) Canonical processes of semantically annotated media production. Multimedia Syst 14(6):327-340. https://doi.org/10.1007 /s00530-008-0134-0
  19. Kim S, Han J (2014) Text of white paper on mpeg-v. Tech. Rep ISO/IEC JTC 1/SC 29/WG 11 W14187, San Jose, USA
  20. Kim SK (2013) Authoring multisensorial content. Signal Process Image Commun 28(2):162-167. https://doi.org/10.1016/j.image.2012.10.011
  21. Kim SK, Yang SJ, Ahn CH, Joo YS (2014) Sensorial information extraction and mapping to generate temperature sensory effects. ETRI J 36(2):224-231. https://doi.org/10.4218/etrij.14.2113.0065
  22. Lasecki W, Miller C, Sadilek A, Abumoussa A, Borrello D, Kushalnagar R, Bigham J (2012) Real- time captioning by groups of non-experts. In: Proceedings of the 25th annual ACM symposium on User interface software and technology -UIST '12, UIST '12. ACM Press, New York, pp 23-33. https://doi.org/10.1145/2380116.2380122
  23. Masiar A, Simko J (2015) Short video metadata acquisition game. In: 10th international work- shop on semantic and social media adaptation and personalization (SMAP). IEEE, pp 61-65. https://doi.org/10.1109/SMAP.2015.7370092
  24. McNaney R, Othman M, Richardson D, Dunphy P, Amaral T, Miller N, Stringer H, Olivier P, Vines J (2016) Speeching: mobile crowdsourced speech assessment to support self-monitoring and management for people with parkinson's. In: Proceedings of the 2016 CHI conference on human factors in computing sys -CHI '16, CHI '16. ACM Press, New York, pp 4464-4476. https://doi.org/10.1145/2858036.2858321
  25. Murray N, Lee B, Qiao Y, Muntean GM (2016) The influence of human factors on olfaction based mulsemedia quality of experience. https://doi.org/10.1109/QoMEX.2016.7498975
  26. Neto FRA, Santos CAS (2018) Understanding crowdsourcing projects: a systematic review of tenden- cies, workflow, and quality management. Inf Process Manag 54(4):490-506. https://doi.org/10.1016 /j.ipm.2018.03.006
  27. Oh HW, Huh JD (2017) Auto generation system of mpeg-v motion sensory effects based on media scene. In: 2017 IEEE international conference on consumer electronics (ICCE). IEEE, pp 160-163. https://doi.org/10.1109/ICCE.2017.7889269
  28. Rainer B, Waltl M, Cheng E, Shujau M, Timmerer C, Davis S, Burnett I, Ritz C, Hellwagner H (2012) Investigating the impact of sensory effects on the quality of experience and emotional response in web videos. In: 4th international workshop on quality of multimedia experience (qoMEX). IEEE, pp 278- 283. https://doi.org/10.1109/QoMEX.2012.6263842
  29. Sadallah M, Aubert O, Prié Y (2014) Chm: an annotation-and component-based hypervideo model for the web. Multimed Tools Appl 70(2):869-903. https://doi.org/10.1007/s11042-012-1177-y
  30. Saleme EB, Celestrini JR, Santos CAS (2017) Time evaluation for the integration of a gestural interactive application with a distributed mulsemedia platform. In: Proceedings of the 8th ACM on multimedia sys- tems conference, MMSys'17. ACM, New York, pp 308-314. https://doi.org/10.1145/3083187.3084013
  31. Saleme EB, Santos CAS, Ghinea G (2018) Coping with the challenges of delivering multiple sensorial media. IEEE MultiMedia, 1-1. https://doi.org/10.1109/MMUL.2018.2873565
  32. Shin SH, Ha KS, Yun HO, Nam YS (2016) Realistic media authoring tool based on mpeg-v interna- tional standard. In: 2016 8th international conference on ubiquitous and future networks (ICUFN). IEEE, pp 730-732. https://doi.org/10.1109/ICUFN.2016.7537133
  33. Taborsky E, Allen K, Blanton A, Jain AK, Klare BF (2015) Annotating unconstrained face imagery: a scalable approach. In: International conference on biometrics (ICB). IEEE, pp 264-271. https://doi.org/10.1109/ICB.2015.7139094
  34. Teki S, Kumar S, Griffiths TD (2016) Large-scale analysis of auditory segregation behavior crowd- sourced via a smartphone app. PLos ONE, 11(4). https://doi.org/10.1371/journal.pone.015
  35. Timmerer C, Waltl M, Rainer B, Hellwagner H (2012) Assessing the quality of sensory experience for multimedia presentations. Signal Process Image Commun 27(8):909-916. https://doi.org/10.101 6/j.image.2012.01.016
  36. van Holthoon F, Olson D (1987) Common sense: the foundations for social science. Common sense. University Press of America, Lanham
  37. Waltl M, Rainer B, Timmerer C, Hellwagner H (2013) An end-to-end tool chain for sensory expe- rience based on mpeg-v. Signal Process Image Commun 28(2):136-150. https://doi.org/10.1016 /j.image.2012.10.009
  38. Waltl M, Timmerer C, Hellwagner H (2010) Improving the quality of multimedia experience through sensory effects. In: Second international workshop on quality of multimedia experience (qoMEX). IEEE, pp 124-129
  39. Waltl M, Timmerer C, Rainer B, Hellwagner H (2012) Sensory effect dataset and test setups. In: 4th international workshop on quality of multimedia experience (qoMEX). IEEE, pp 115-120. https://doi.org/10.1109/QoMEX.2012.6263841
  40. Yuan Z, Bi T, Muntean GM, Ghinea G (2015) Perceived synchronization of mulsemedia services. IEEE Trans Multimedia 17(7):957-966. https://doi.org/10.1109/TMM.2015.2431915
  41. Yue T, Wang H, Cheng S (2018) Learning from users: a data-driven method of qoe evaluation for internet video. MTAP: Multimedia Tools Appl, 1-32. https://doi.org/10.1007/s11042-018-5918-4
  42. Zhai H, Lingren T, Deleger L, Li Q, Kaiser M, Stoutenborough L, Solti I (2013) Web 2.0-based crowd- sourcing for high-quality gold standard development in clinical natural language processing. J Med Internet Res 15(4):1-17. https://doi.org/10.2196/jmir.2426
  43. Marcello Novaes de Amorim is a doctorate candidate in the Computer Science Department at Federal Uni- versity of Espírito Santo, Brazil. He received the graduation degree in Computer Science from UFES, Brazil, in 2005, and the M.Sc. degree in Computer Science from the Federal University of Espírito Santo, Brazil, in 2007. His research interests include multimedia systems, human computation and crowdsourcing. Contact him at novaes@inf.ufes.br.
  44. Est êv ão Bissoli Saleme is currently Ph.D. candidate in the Computer Science at Federal University of Espírito Santo (UFES), Brazil. From August 2018 and Februay 2019, he was an Academic Visitor at Brunel University London, UK. He received the B.Sc. degree in Information Systems from FAESA, Brazil, in 2008, and the M.Sc. degree in Computer Science from the UFES, in 2015. His current research interests include multimedia/mulsemedia systems, middlewares and frameworks, interactive multimedia, media transport and delivery. Contact him at estevaobissoli@gmail.com.
  45. F ábio Ribeiro de Assis Neto earned the B.Sc. and M.Sc. degrees in Computer Science from the Federal University of Espírito Santo (UFES), Brazil, in 2012 and 2017, respectively. His research interests include crowdsourcing, multimedia systems, and human computation. Contact him at fabio.ribeiro.neto@gmail.com.
  46. Dr. Celso A. S. Santos is a Professor in the Department of Informatics at Federal University of Espírito Santo (UFES), Brazil. He received the B.S. degree in Electrical Engineering from UFES in 1991, and the M.S. degree in Electrical Engineering (Electronic Systems) from the Polytechnic School of the Univer- sity of São Paulo, Brazil, in 1994. In 1999, he received his Dr. degree at Informatique Fondamentalle et Parallelisme from Universitè Paul Sabatier de Toulouse III, France. His recent research interests focus on multimedia/mulsemedia systems and applications, synchronization, and crowdsourcing systems. Contact him at saibel@inf.ufes.br.

Related papers

MadPad: A Crowdsourcing System for Audiovisual Sampling
Ge Wang

2011

MadPad is a networked audiovisual sample station for mobile devices. Twelve short video clips are loaded onto the screen in a grid and playback is triggered by tapping anywhere on the clip. This is similar to tapping the pads of an audio sample station, but extends that interaction to add visual sampling. Clips can be shot on-the-y with a cameraenabled mobile device and loaded into the player instantly, giving the performer an ability to quickly transform his or her surroundings into a sample-based, audiovisual instrument. Samples can also be sourced from an online community in which users can post or download content. The recent ubiquity of multitouch mobile devices and advances in pervasive computing have made this system possible, providing for a vast amount of content only limited by the imagination of the performer and the community. This paper presents the core features of MadPad and the design explorations that inspired them.

View PDFchevron_right
Crowdsourcing-based multimedia subjective evaluations
T. Hoßfeld, Judith Redi, Pavel Korshunov

Proceedings of the 2nd ACM international workshop on Crowdsourcing for multimedia - CrowdMM '13, 2013

Research on Quality of Experience (QoE) heavily relies on subjective evaluations of media. An important aspect of QoE concerns modeling and quantifying the subjective notions of 'beauty' (aesthetic appeal) and 'something well-known' (content recognizability), which are both subject to cultural and social effects. Crowdsourcing, which allows employing people worldwide to perform short and simple tasks via online platforms, can be a great tool for performing subjective studies in a time and cost-effective way. On the other hand, the crowdsourcing environment does not allow for the degree of experimental control which is necessary to guarantee reliable subjective data. To validate the use of crowdsourcing for QoE assessments, in this paper, we evaluate aesthetic appeal and recognizability of images using the Microworkers crowdsourcing platform and compare the outcomes with more conventional evaluations conducted in a controlled lab environment. We find high correlation between crowdsourcing and lab scores for recognizability but not for aesthetic appeal, indicating that crowdsourcing can be used for QoE subjective assessments as long as the workers' tasks are designed with extreme care to avoid misinterpretations.

View PDFchevron_right
Integrating Crowdsourcing and Human Computation for Complex Video Annotation Tasks
Celso A S Santos

2020

Video annotation is an activity that aims to supplement this type of multimedia object with additional content or information about its context, nature, content, quality and other aspects. These annotations are the basis for building a variety of multimedia applications for various purposes ranging from entertainment to security. Manual annotation is a strategy that uses the intelligence and workforce of people in the annotation process and is an alternative to cases where automatic methods cannot be applied. However, manual video annotation can be a costly process because as the content to be annotated increases, so does the workload for annotating. Crowdsourcing appears as a viable solution strategy in this context because it relies on outsourcing the tasks to a multitude of workers, who perform specific parts of the work in a distributed way. However, as the complexity of required media annoyances increases, it becomes necessary to employ skilled labor, or willing In: II Concurso...

View PDFchevron_right
Multimedia content creation using societal-scale ubiquitous camera networks and human-centric wearable sensing
Nanwei Gong

Proceedings of the 18th ACM international conference on Multimedia, 2010

We present a novel approach to the creation of user-generated, documentary video using a distributed network of sensorenabled video cameras and wearable on-body sensor devices. The wearable sensors are used to identify the subjects in view of the camera system and label the captured video with real-time human-centric social and physical behavioral information. With these labels, massive amounts of continually recorded video can be browsed, searched, and automatically stitched into cohesive multimedia content. This system enables naturally occurring human behavior to drive and control a multimedia content creation system in order to create video output that is understandable, informative, and/or enjoyable to its human audience. The collected sensor data is further utilized to enhance the created multimedia content such as by using the data to edit and/or generate audio score, determine appropriate pacing of edits, and control the length and type of audio and video transitions directly from the content of the captured media. We present the design of the platform, the design of the multimedia content creation application, and the evaluated results from several live runs of the complete system.

View PDFchevron_right
On the role of user-generated metadata in audio visual collections
Michiel Hildebrand

Proceedings of the …, 2011

Recently, various crowdsourcing initiatives showed that targeted efforts of user communities result in massive amounts of tags. For example, the Netherlands Institute for Sound and Vision collected a large number of tags with the video labeling game Waisda?. To successfully utilize these tags, a better understanding of their characteristics is required. The goal of this paper is twofold: (i) to investigate the vocabulary that users employ when describing videos and compare it to the vocabularies used by professionals; and (ii) to establish which aspects of the video are typically described and what type of tags are used for this. We report on an analysis of the tags collected with Waisda?. With respect to the first goal, we compared the the tags with a typical domain thesaurus used by professionals, as well as with a more general vocabulary. With respect to the second goal, we compare the tags to the video subtitles to determine how many tags are derived from the audio signal. In addition, we perform a qualitative study in which a tag sample is interpreted in terms of an existing annotation classification framework. The results suggest that the tags complement the metadata provided by professional cataloguers, the tags describe both the audio and the visual aspects of the video, and the users primarily describe objects in the video using general descriptions.

View PDFchevron_right

Related topics

  • Information Systems
  • Computer Science
  • Multimedia Authoring
  • Multimedia Annotation
  • Sensory Effects
  • Mulsemedia content
    • 580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved