Multimodal sensing-based camera applications

Sensors, 2016

Camera-based interfaces in mobile devices are starting to be used in games and apps, but few works have evaluated them in terms of usability or user perception. Due to the changing nature of mobile contexts, this evaluation requires extensive studies to consider the full spectrum of potential users and contexts. However, previous works usually evaluate these interfaces in controlled environments such as laboratory conditions, therefore, the findings cannot be generalized to real users and real contexts. In this work, we present a robust camera-based interface for mobile entertainment. The interface detects and tracks the user's head by processing the frames provided by the mobile device's front camera, and its position is then used to interact with the mobile apps. First, we evaluate the interface as a pointing device to study its accuracy, and different factors to configure such as the gain or the device's orientation, as well as the optimal target size for the interface. Second, we present an in the wild study to evaluate the usage and the user's perception when playing a game controlled by head motion. Finally, the game is published in an application store to make it available to a large number of potential users and contexts and we register usage data. Results show the feasibility of using this robust camera-based interface for mobile entertainment in different contexts and by different people.

Society of Photo-Optical …, 2008

The video applications on mobile communication devices have usually been designed for content creation, access, and playback. For instance, many recent mobile devices replicate the functionalities of portable video cameras and video recorders, and digital TV receivers. These are all demanding uses, but nothing new from the consumer point of view. However, many of the current devices have two cameras built in, one for capturing high resolution images, and the other for lower, typically VGA (640x480 pixels) resolution video telephony. We employ video to enable new applications and describe four actual solutions implemented on mobile communication devices. The first one is a real-time motion based user interface that can be used for browsing large images or documents such as maps on small screens. The motion information is extracted from the image sequence captured by the camera. The second solution is a real-time panorama builder, while the third one assembles document panoramas, both from individual video frames. The fourth solution is a real-time face and eye detector. It provides another type of foundation for motion based user interfaces as knowledge of presence and motion of a human faces in the view of the camera can be a powerful application enabler.

2005

In mobile devices, such as mobile phones and PDAs, an integrated camera can be used to interact with the device in new ways. In this paper we introduce the term mixed interaction space and argue that the possibility of using mixed interaction spaces is what distinguishes camerabased interaction from other types of sensor-based interaction on mobile devices. We present our implemented applications, and related work that use mixed interaction spaces. Based on this we address how mixed interaction spaces can have different identities, be mapped to applications, and how it can be visualized.

Proceedings of the 19th annual ACM symposium on User interface software and technology - UIST '06, 2006

This paper presents TinyMotion, a pure software approach for detecting a mobile phone user's hand movement in real time by analyzing image sequences captured by the built-in camera. We present the design and implementation of Ti-nyMotion and several interactive applications based on TinyMotion. Through both an informal evaluation and a formal 17-participant user study, we found that 1. TinyMotion can detect camera movement reliably under most background and illumination conditions. 2. Target acquisition tasks based on TinyMotion follow Fitts' law and Fitts' law parameters can be used for TinyMotion based pointing performance measurement. 3. The users can use Vision Tilt-Text, a TinyMotion enabled input method, to enter sentences faster than MultiTap with a few minutes of practicing. 4. Using camera phone as a handwriting capture device and performing large vocabulary, multilingual real time handwriting recognition on the cell phone are feasible. 5. TinyMotion based gaming is enjoyable and immediately available for the current generation camera phones. We also report user experiences and problems with TinyMotion based interaction as resources for future design and development of mobile interfaces.

2008

In recent years mobile devices have been deployed with various new technologies, such as high quality cameras and the ability to support rich multimedia. Vision based technology such as motion detection has, until recently been limited to more powerful desktop devices. This paper lays out a brief review of these technologies with the aim of introducing the concept of vision based interaction on mobile devices and substantiate an implementation thereof. We conclude that mobile phones are currently being deployed with hardware and software which can support vision based interactions and which in the future could be widely deployed. We predict that in the near future vision based interactions such as gesture control will become prevalent and greatly enhance mobile devices.

2010

Abstract In the ubiquitous computing environment, people will interact with everyday objects (or computers embedded in them) in ways different from the usual and familiar desktop user interface. One such typical situation is interacting with applications through large displays such as televisions, mirror displays, and public kiosks. With these applications, the use of the usual keyboard and mouse input is not usually viable (for practical reasons). In this setting, the mobile phone has emerged as an excellent device for novel interaction.

This paper presents a study about consumers requirements of mobile applications. More specifically, mobile camera related uses. An application was developed because Mobile Phone Camera users demanded a faster photo taking facility, especially while user requires use of mobile camera when they are moving. The paper also presents how and why people adapted the application in their daily lives. The application when launched will start taking photos automatically in the Burst mode (capturing a series of photos one after the other) until the user manually stops it.

2004

The idea described in this paper is to use the built-in cameras of consumer mobile phones as sensors for 2-dimensional visual codes. Such codes can be attached to physical objects in order to retrieve object-related information and functionality. They are also suitable for display on electronic screens. The proposed visual code system allows the simultaneous detection of multiple codes, introduces a position-independent coordinate system, and provides the phone's orientation as a parameter. The ability to detect objects in the user's vicinity offers a natural way of interaction and strengthens the role of mobile phones in a large number of application scenarios. We describe the hardware requirements, the design of a suitable visual code, a lightweight recognition algorithm, and present some example applications.

1996

In the future, the computer will be thought of more as an assistant than as a tool, and users will increasingly expect machines to make decisions on their behalf. As with a human assistant, a machine’s ability to make informed choices will often depend on the extent of its knowledge of activities in the world around it. Equipping personal computers with a large number of sensors for monitoring their environment is, however, expensive and inconvenient, and a preferable solution would involve a small number of input devices with a broad scope of application. Video cameras are ideally suited to many real- world monitoring applications for this reason. In addition, recent reductions in the manufacturing costs of simple cameras will soon make their widespread deployment in the home and office economically viable. The use of video as an input device also allows the creation of new types of user-interface, more suitable in some circumstances than those afforded by the conventional keyboard and mouse. This thesis examines some examples of these ‘Video-Augmented Environments’ and related work, and then describes two applications in detail. The first, a ‘software cameraman’, uses the analysis of one video stream to control the display of another. The second, ‘BrightBoard’, allows a user to control a computer by making marks on a conventional whiteboard, thus ‘augmenting’ the board with many of the facilities common to electronic documents, including the ability to fax, save, print and email the image of the board. The techniques which were found to be useful in the construction of these applications are common to many systems which monitor real-world video, and so they were combined in a toolkit called ‘Vicar’. This provides an architecture for ‘video plumbing’, which allows standard video- processing components to be connected together under the control of a scripting language. It is a single application which can be programmed to create a variety of simple Video-Augmented Environments, such as those described above, without the need for any recompilation, and so should simplify the construction of such applications in the future. Finally, opportunities for further exploration on this theme are discussed.

2009 International Symposium on Ubiquitous Virtual Reality, 2009

Smart phones are becoming visual computing powerhouses. Using sensors such as camera, GPS, and others, the device can provide a new user interface to the real world, augmenting user's view of the world with additional information and controls. Combining computation with image capture allows new kind of photography that can be more expressive than is possible to obtain with a traditional camera. New APIs allow harnessing more computation power of a smart phone to visual processing than what one can obtain from just a CPU.