Camera-Based Virtual Environment Interaction on Mobile Devices

International Journal of Interactive Multimedia and Artificial Intelligence, 2019

Three Dimensional (3D) interaction is the plausible human interaction inside a Virtual Environment (VE). The rise of the Virtual Reality (VR) applications in various domains demands for a feasible 3D interface. Ensuring immersivity in a virtual space, this paper presents an interaction technique where manipulation is performed by the perceptive gestures of the two dominant fingers; thumb and index. The two fingertip-thimbles made of paper are used to trace states and positions of the fingers by an ordinary camera. Based on the positions of the fingers, the basic interaction tasks; selection, scaling, rotation, translation and navigation are performed by intuitive gestures of the fingers. Without keeping a gestural database, the features-free detection of the fingers guarantees speedier interactions. Moreover, the system is user-independent and depends neither on the size nor on the color of the users' hand. With a case-study project; Interactions by the Gestures of Fingers (IGF) the technique is implemented for evaluation. The IGF application traces gestures of the fingers using the libraries of OpenCV at the back-end. At the front-end, the objects of the VE are rendered accordingly using the Open Graphics Library; OpenGL. The system is assessed in a moderate lighting condition by a group of 15 users. Furthermore, usability of the technique is investigated in games. Outcomes of the evaluations revealed that the approach is suitable for VR applications both in terms of cost and accuracy.

The user interaction in mobile phones has been traditionally based on key behaviors that users have adapted around their mobile devices. Although this type of interaction could be suitable for some contents, it could be improved from the point of view of the usability. In this paper we present a new user interface based on 3D navigation oriented to mobile phones. To enhance the user experience, we have developed a new interface based on 3D real-time graphics where users interact by means of natural movements of their devices. The goals of this development consist in enhancing the user interaction and accessibility to web content or interactive multimedia applications by means of avoiding a key-based, or a mouse-based, navigation and proposing a software solution adaptable to multiple and different mobile devices. In this sense, the user inputs can be detected by different input devices such as accelerometers or cameras, as well as the traditional keypads. Since mobile phones with on-board digital cameras are now widely available at low cost, the proposed 3D user interface exploits the acquisition capabilities of these input devices. In this sense, a differential algorithm has been applied in order to estimate phone movements from video images. The results of the performance evaluation of the 3D user interface shows that the proposed algorithm not only obtains a motion and tracking under extreme lighting conditions, but also adds an insignificant overhead to the system performance. Finally, a 3D environment has been designed to evaluate the performance of the presented approach, which has been successfully tested in actual users.

Technology, 2008

Modern computer games often explore graphics quality while leaving interaction and gameplay behind. Arcade machines may improve gamer experience by using special devices (instead of traditional joysticks) to interact with the game. We propose a technique for 3D "point-and-click" interaction that separates both camera and pointer control. It also explores audio and visual collision detection feedback to improve user spatial sense. The experiments were done within a virtual 3D environment, similar to that used in first person shooter games (FPS), a genre popular and well known by both the entertainment industry and gamers. It can also be used in applications other than games, to perform tasks such as "drag-and-dropping" objects, drawing and selecting in general. The results were compared with the traditional FPS interaction technique (mouse-driven camera with point-and-click interaction).

2000

This paper proposes a new and natural human computer interface for interacting with virtual environments. The 3D pointing direction of a user in a virtual environment is estimated using monocular computer vision. The 2D position of the user' s hand is extracted in the image plane and then mapped to a 3D direction using knowledge about the position of the

International Journal of Advancements in Computing Technology, 2012

Normally all digital images are display in a flat twodimensional (2D) image space. However, the three dimensional (3D) techniques exist that allow users to interact and walk through a virtual environment in mobile device using openglES. S o in this paper, an interactive virtual three dimensional (3D) photo gallery on mobile device is presented. The system will allow users to take pictures with the mobile device and exhibit in the form of virtual 3D gallery and navigate or walk through in the gallery by pressing the button or moving the device. The device has the computational unit which is capable of determining motion data from the g-sensor or accelerometer sensor. The motion data describes the movement of the device including a rotation of the device. Therefore, the benefit of the sensor could be applied such as modifying the view displayed on the screen vertically or horizontally automatically. Also the system can detect tilting leftright device for viewing image and navigating in the virtual 3D Gallery as well. The result shows that the interactive virtual 3D photo gallery contains digital photos taken by a user and the user can navigates through the gallery by pressing buttons or moving the mobile device.

Proceedings of the 2018 International Conference on Advanced Visual Interfaces, 2018

Gamepads and 3D controllers are the main controllers used in most Virtual Environments. Despite being simple to use, these input devices have a number of limitations as fixed layout and difficulty to remember the mapping between buttons and functions. Mobile devices present interesting characteristics that might be valuable in immersive environments: more flexible interfaces, touchscreen combined with onboard sensors that allow new interaction and easy acceptance since these devices are used daily by most users. The work described in this article proposes a solution that uses mobile devices to interact with Immersive Virtual Environments for selection and navigation tasks. The proposed solution uses the mobile device camera to track the Head-Mounted-Display position and present a virtual representation of the mobile device screen; it was tested using an Immersive Virtual Museum as use case. Based on this prototype, a study was performed to compare controller based and mobile based interaction for navigation and selection showing that using mobile devices is viable in this context and offers interesting interaction opportunities.

2008

This paper presents a six degrees of freedom (6 DoF) optical tracking device for interaction in desktop VR applications. The device uses three webcams mounted on a box and uses computer vision techniques to detect the movements of white markers over a black background. We present the process of movements' detection and how this information is adapted to be applied in interaction events and to simulate a 3D mouse device.

1996

In the future, the computer will be thought of more as an assistant than as a tool, and users will increasingly expect machines to make decisions on their behalf. As with a human assistant, a machine’s ability to make informed choices will often depend on the extent of its knowledge of activities in the world around it. Equipping personal computers with a large number of sensors for monitoring their environment is, however, expensive and inconvenient, and a preferable solution would involve a small number of input devices with a broad scope of application. Video cameras are ideally suited to many real- world monitoring applications for this reason. In addition, recent reductions in the manufacturing costs of simple cameras will soon make their widespread deployment in the home and office economically viable. The use of video as an input device also allows the creation of new types of user-interface, more suitable in some circumstances than those afforded by the conventional keyboard and mouse. This thesis examines some examples of these ‘Video-Augmented Environments’ and related work, and then describes two applications in detail. The first, a ‘software cameraman’, uses the analysis of one video stream to control the display of another. The second, ‘BrightBoard’, allows a user to control a computer by making marks on a conventional whiteboard, thus ‘augmenting’ the board with many of the facilities common to electronic documents, including the ability to fax, save, print and email the image of the board. The techniques which were found to be useful in the construction of these applications are common to many systems which monitor real-world video, and so they were combined in a toolkit called ‘Vicar’. This provides an architecture for ‘video plumbing’, which allows standard video- processing components to be connected together under the control of a scripting language. It is a single application which can be programmed to create a variety of simple Video-Augmented Environments, such as those described above, without the need for any recompilation, and so should simplify the construction of such applications in the future. Finally, opportunities for further exploration on this theme are discussed.

1996

This paper outlines a system design and implementation of a 3D input device for graphical applications. It is shown how computer vision can be used to track a users movements within the image frame allowing interaction with 3D worlds and objects. Point Distribution Models (PDMs) have been shown to be successful at tracking deformable objects. This system demonstrates how these ‘smart snakes’ can be used in real time with real world applications, demonstrating how computer vision can provide a low cost, intuitive interface that has few hardware constraints. The compact mathematical model behind the PDM allows simple static gesture recognition to be performed providing the means to communicate with an application. It is shown how movement of both the hand and face can be used to drive 3D engines. The system is based upon Open Inventor and designed for use with Silicon Graphics Indy Workstations but allowances have been made to facilitate the inclusion of the tracker within third party...