FingerCloud

In this work, we present a fingertip tracking framework which allows observation of finger movements in task space. By applying a multi-scale edge extraction technique, an edge map is generated in which low contrast edges are preserved while noise is suppressed. Based on circular image features, determined from the map using Hough transform, the fingertips are accurately tracked by combining a particle filter and a subsequent mean-shift procedure. To increase the robustness of the proposed method, dynamical motion models are trained for the prediction of the finger displacements. Experiments were conducted on various image sequences from which statements on the performance of the framework can be derived.

We present a fully disclosed tracking pipeline based on mutual capacitance sensing. We designed both hard-and software to support and foster experimentation. Without the use of vendor-supplied cap-sense ICs and proprietary software parts, every aspect and parameter of the user interface can be modified with ease. This flexibility gives the researcher a high degree of freedom while designing novel interactive systems. The working prototype of an audio-tactile multi-touch map is presented as an exemplary application.

2008

We present a flexible system to track the movements of a bare finger on a flat surface. The proposed system is able to discriminate whether the user is touching or just pointing at the surface. The system works using two webcams and a fast scanline-based algorithm. The initial setup of the two webcams is easy and fast. No markers, gloves, or other hand-held devices are required. Since the system is independent from the nature of the pointing surface, it is possible to use a screen or a projected wall as a virtual touchscreen. The complexity of the algorithms used by the system grows less than linearly with resolution, making the software layer very lightweight and suitable also for low-powered devices like embedded controllers.

Sensors

Rings are widely accepted wearables for gesture interaction. However, most rings can sense only the motion of one finger or the whole hand. We present PeriSense, a ring-shaped interaction device enabling multi-finger gesture interaction. Gestures of the finger wearing ring and its adjacent fingers are sensed by measuring capacitive proximity between electrodes and human skin. Our main contribution is the determination of PeriSense’s interaction space involving the evaluation of capabilities and limitations. We introduce a prototype named PeriSense, analyze the sensor resolution at different distances, and evaluate finger gestures and unistroke gestures based on gesture sets allowing the determination of the strengths and limitations. We show that PeriSense is able to sense the change of conductive objects reliably up to 2.5 cm. Furthermore, we show that this capability enables different interaction techniques such as multi-finger gesture recognition or two-handed unistroke input.

User Interface Conference, 2000. …, 2000

A trend in computing environments today is to move towards more 'natural' interaction, another is to make hardware invisible to the user. Both these ideas converge into ubiquitous computing -the Digital Desk is an example of this idea. In this paper we concentrate on an input device for the Digital Desk, namely the user's fingertip, which is made to act like a mouse. Tracking such an input device is common to a number of augmented reality environments and involves vision and motion analysis.

IEEE Computer Graphics and Applications, 2002

A ugmented desk interfaces and other virtual reality systems depend on accurate, real-time hand and fingertip tracking for seamless integration between real objects and associated digital information. We introduce a method for discerning fingertip locations in image frames and measuring fingertips trajectories across image frames. We also propose a mechanism for combining direct manipulation and symbolic gestures based on multiple fingertip motions. Our method uses a filtering technique, in addition to detecting fingertips in each image frame, to predict fingertip locations in successive image frames and to examine the correspondences between the predicted locations and detected fingertips. This lets us obtain multiple fingertips' trajectories in real time and improves fingertip tracking. This method can track multiple fingertips reliably even on a complex background under changing lighting conditions without invasive devices or color markers. Distinguishing the thumb lets us differentiate manipulative (extended thumb) from symbolic (folded thumb) gestures. We base this on the observation that users generally use only a thumb and forefinger in fine manipulation. The method then uses the Hidden Markov Model (HMM), 1 which interprets hand and finger motions as symbolic events based on a probabilistic framework, to recognize symbolic gestures for application to interactive systems. Other researchers have used HMM to recognize body, hand, and finger motions. 2,3 Augmented desk interfaces Several augmented desk interface systems have been developed recently. 4,5 One of the earliest attempts in this domain, DigitalDesk, 6 uses a charge-coupled device

Since the advent of multitouch screens users have been able to interact using fingertip gestures in a two dimensional plane. With the development of depth cameras, such as the Kinect, attempts have been made to reproduce the detection of gestures for three dimensional interaction. Many of these use contour analysis to find the fingertips, however the success of such approaches is limited due to sensor noise and rapid movements. This paper discusses an approach to identify fingertips during rapid movement at varying depths allowing multitouch without contact with a screen. To achieve this, we use a weighted graph that is built using the depth information of the hand to determine the geodesic maxima of the surface. Fingertips are then selected from these maxima using a simplified model of the hand and correspondence found over successive frames. Our experiments show real-time performance for multiple users providing tracking at 30fps for up to 4 hands and we compare our results with stateof-the-art methods, providing accuracy an order of magnitude better than existing approaches.

Proceedings of the ACM on Human-Computer Interaction

Touch sensing on ad-hoc surfaces has the potential to transform everyday surfaces in the environment - desks, tables and walls - into tactile, touch-interactive surfaces, creating large, comfortable interactive spaces without the cost of large touch sensors. Depth sensors are a promising way to provide touch sensing on arbitrary surfaces, but past systems have suffered from high latency and poor touch detection accuracy. We apply a novel state machine-based approach to analyzing touch events, combined with a machine-learning approach to predictively classify touch events from depth data with lower latency and higher touch accuracy than previous approaches. Our system can reduce end-to-end touch latency to under 70ms, comparable to conventional capacitive touchscreens. Additionally, we open-source our dataset of over 30,000 touch events recorded in depth, infrared and RGB for the benefit of future researchers.

arXiv (Cornell University), 2023

Automated hand gesture recognition has been a focus of the AI community for decades. Traditionally, work in this domain revolved largely around scenarios assuming the availability of the flow of images of the operator's/user's hands. This has partly been due to the prevalence of camera-based devices and the wide availability of image data. However, there is growing demand for gesture recognition technology that can be implemented on low-power devices using limited sensor data instead of high-dimensional inputs like hand images. In this work, we demonstrate a hand gesture recognition system and method that uses signals from capacitive sensors embedded into the etee hand controller. The controller generates real-time signals from each of the wearer's five fingers. We use a machine learning technique to analyse the time-series signals and identify three features that can represent 5 fingers within 500 ms. The analysis is composed of a two-stage training strategy, including dimension reduction through principal component analysis and classification with K-nearest neighbour. Remarkably, we found that this combination showed a level of performance which was comparable to more advanced methods such as supervised variational autoencoder. The base system can also be equipped with the capability to learn from occasional errors by providing it with an additional adaptive error correction mechanism. The results showed that the error corrector improve the classification performance in the base system without compromising its performance. The system requires no more than 1 ms of computing time per input sample, and is smaller than deep neural networks, demonstrating the feasibility of agile gesture recognition systems based on this technology.

2013

We present FingerPad, a nail-mounted device that turns the tip of the index finger into a touchpad, allowing private and subtle interaction while on the move. FingerPad enables touch input using magnetic tracking, by adding a Hall sensor grid on the index fingernail, and a magnet on the thumbnail. Since it permits input through the pinch gesture, Fin-gerPad is suitable for private use because the movements of the fingers in a pinch are subtle and are naturally hidden by the hand. Functionally, FingerPad resembles a touchpad, and also allows for eyes-free use. Additionally, since the necessary devices are attached to the nails, FingerPad preserves natural haptic feedback without affecting the native function of the fingertips. Through user study, we analyze the three design factors, namely posture, commitment method and target size, to assess the design of the FingerPad. Though the results show some trade-off among the factors, generally participants achieve 93% accuracy for very small targets (1.2mm-width) in the seated condition, and 92% accuracy for 2.5mm-width targets in the walking condition.