Mulsemedia systems

Designing a mulsemedia-multiple sensorial media-system entails first and foremost comprehending what it is beyond the ordinary understanding that it engages users in digital multisensory experiences that stimulate other senses in addition to sight and hearing, such as smell, touch, and taste. A myriad of programs that comprise a software system, several output devices to deliver sensory effects, computer media, among others, dwell deep in the realm of mulsemedia systems, making it a complex task for newcomers to get acquainted with their concepts and terms. Although there have been many technological advances in this field, especially for multisensory devices, there is a shortage of work that tries to establish common ground in terms of formal and explicit representation of what mulsemedia systems encompass. This might be useful to avoid the design of feeble mulsemedia systems that can be barely reused owing to misconception. In this paper, we extend our previous work by proposing to establish a common conceptualization about mulsemedia systems through a domain reference ontology named MulseOnto to aid the design of them. We applied ontology verification and validation techniques to evaluate it, including assessment by humans and a data-driven approach whereby the outcome is three successful instantiations of MulseOnto for distinct cases, making evident its ability to accommodate heterogeneous mulsemedia scenarios.

The use of multiple senses in interactive applications has become increasingly feasible due to the upsurge of commercial, off-the-shelf devices to produce sensory effects. Creating Multiple Sensorial Media (MulSeMedia) immersive systems requires understanding their digital ecosystem. Mulsemedia systems encompass a set of applications, and devices of different types assembled to communicate or express feelings from the virtual world to the real world. Despite existing standards, tools, and recent research devoted to them, there is still a lack of formal and explicit representation of what mulsemedia is. Misconceptions could eventually lead to the construction of solutions that might not take into account reuse, integration, standardization, among other design features. In this paper, we propose to establish a common conceptualization about mulsemedia systems through a reference ontology, named MulseOnto, covering their main notions. To evaluate it, we applied ontology verification and validation techniques, including assessment by humans and a data-driven approach. The results showed that MulseOnto can be used as a consensual conceptual model for exploring the knowledge about the whole chain of mulsemedia systems. CCS CONCEPTS • General and reference → Reference works; • Information systems → Multimedia information systems; • Human-centered computing → Ubiquitous and mobile computing theory, concepts and paradigms;

Technological advances in computing have allowed multimedia systems to create more immersive experiences for users. Beyond the traditional senses of sight and hearing, researchers have observed that the use of smell, taste, and touch in such systems is becoming increasingly well-received, leading to a new category of multimedia systems called mulsemedia-multiple sensorial media-systems. In parallel, these systems introduce heterogeneous technologies to deliver different sensory effects such as lighting, wind, vibration, and smell, under varied conditions and restrictions. This new paradigm shift poses many challenges, mainly related to mulsemedia integration, delay, responsiveness, sensory effects intensities, wearable and other heterogeneous devices for delivering sensory effects, and remote delivery of mulsemedia components. In addition, new approaches to interacting with multimedia applications have emerged such as multi-touch interfaces, voice processing, and brain-computer interfaces, giving rise to new kinds of complex interactive systems. In this article, we underpin fundamental challenges for delivering multisensory effects to heterogeneous systems. We propose an interoperable mulsemedia framework for coping with these challenges, meeting the emerging requirements. It is achieved through the evolution of an open distributed mulsemedia system. We changed its core following architectural and design patterns to accommodate different profiles of communication, connectivity, and sensory effects metadata standard according to the need of mulsemedia applications and devices available in the user's environment. The results include case studies where the framework has been duly applied.

Technological advances in computing have allowed multimedia systems to create more immersive experiences for users. Beyond the traditional senses of sight and hearing, researchers have observed that the use of smell, taste, and touch in such systems is becoming increasingly well-received, leading to a new category of multimedia systems called mulsemedia-multiple sensorial media-systems. In parallel, these systems introduce heterogeneous technologies to deliver different sensory effects such as lighting, wind, vibration, and smell, under varied conditions and restrictions. This new paradigm shift poses many challenges, mainly related to mulsemedia integration, delay, responsiveness, sensory effects intensities, wearable and other heterogeneous devices for delivering sensory effects, and remote delivery of mulsemedia components. In addition, new approaches to interacting with multimedia applications have emerged such as multi-touch interfaces, voice processing, and brain-computer interfaces, giving rise to new kinds of complex interactive systems. In this article, we underpin fundamental challenges for delivering multisensory effects to heterogeneous systems. We propose an interoperable mulsemedia framework for coping with these challenges, meeting the emerging requirements. It is achieved through the evolution of an open distributed mulsemedia system. We changed its core following architectural and design patterns to accommodate different profiles of communication, connectivity, and sensory effects metadata standard according to the need of mulsemedia applications and devices available in the user's environment. The results include case studies where the framework has been duly applied.

The use of multiple senses in interactive applications has become increasingly feasible due to the upsurge of commercial, off-the-shelf devices to produce sensory effects. Creating Multiple Sensorial Media (MulSeMedia) immersive systems requires understanding their digital ecosystem. Mulsemedia systems encompass a set of applications, and devices of different types assembled to communicate or express feelings from the virtual world to the real world. Despite existing standards, tools, and recent research devoted to them, there is still a lack of formal and explicit representation of what mulsemedia is. Misconceptions could eventually lead to the construction of solutions that might not take into account reuse, integration, standardization, among other design features. In this paper, we propose to establish a common conceptualization about mulsemedia systems through a reference ontology, named MulseOnto, covering their main notions. To evaluate it, we applied ontology verification and validation techniques, including assessment by humans and a data-driven approach. The results showed that MulseOnto can be used as a consensual conceptual model for exploring the knowledge about the whole chain of mulsemedia systems. CCS CONCEPTS • General and reference → Reference works; • Information systems → Multimedia information systems; • Human-centered computing → Ubiquitous and mobile computing theory, concepts and paradigms;

One of the main challenges in current multimedia networking environments is to find solutions to help accommodate the next generation of mobile application classes with stringent Quality of Service (QoS) requirements whilst enabling Quality of Experience (QoE) provisioning for users. One such application class, featured in this paper, is 360º mulsemedia-multiple sensorial media-which enriches 360º video by adding sensory effects that stimulate human senses beyond those of sight and hearing, such as the tactile and olfactory ones. In this paper, we present a conceptual framework for 360º mulsemedia delivery and a 360º mulsemedia-based prototype that enables users to experience 360º mulsemedia content. User evaluations revealed that higher video resolutions do not necessarily lead to the highest QoE levels in our experimental setup. Therefore, bandwidth savings can be leveraged with no detrimental impact on QoE. In recent years, innovations in electronics have led to a significant increase in demand for new services and use cases. Future wireless networks are expected to enable ubiquitous connectivity for users, machines, and devices and provide support for an unimaginable exchange of data and multimedia content. In such heterogeneous multiuser , multi-machine, multi-device, multi-technology, and multi-application environments, QoE is foreseen to become the main differentiator between network operators and content providers 1-2 .

Human perception is inherently multisensory involving sight, hearing , smell, touch, and taste. Mulsemedia systems include the combination of traditional media (text, image, video, and audio) with non-traditional ones that stimulate other senses beyond sight and hearing. Whilst work has been done on some user-centred aspects that the distribution of mulsemedia data raises, such as synchro-nisation, and jitter, this paper tackles complementary issues that temporality constraints pose on the distribution of mulsemedia effects. It aims at improving response time interval in networked event-based mulsemedia systems based upon prior findings in this context. Thus, we reshaped the communication strategy of an open distributed mulsemedia platform called PlaySEM to work more efficiently with other event-based applications, such as games, VR/AR software, and interactive applications, wishing to stimulate other senses to increase the immersion of users. Moreover, we added lightweight communic...

Human perception is inherently multisensory involving sight, hearing , smell, touch, and taste. Mulsemedia systems include the combination of traditional media (text, image, video, and audio) with non-traditional ones that stimulate other senses beyond sight and hearing. Whilst work has been done on some user-centred aspects that the distribution of mulsemedia data raises, such as synchro-nisation, and jitter, this paper tackles complementary issues that temporality constraints pose on the distribution of mulsemedia effects. It aims at improving response time interval in networked event-based mulsemedia systems based upon prior findings in this context. Thus, we reshaped the communication strategy of an open distributed mulsemedia platform called PlaySEM to work more efficiently with other event-based applications, such as games, VR/AR software, and interactive applications, wishing to stimulate other senses to increase the immersion of users. Moreover, we added lightweight communic...