Action Anticipation from Multimodal Data

2021

Recent advancement in smart phones and computing technologies has played a vital role in people’s life. Develop a model to detect the human basic dynamic activities such as Amble, Climb stairs, coming down the stairs into the floor and human basic static activities like Sitting, Standing or Laying using the person’s smart phone and computers are the major work of this paper. Conventional Machine learning models like Logistic Regression, SVC, Decision tree, etc. results are compared with a recurrent deep neural network model named as Long Short Term Memory (LSTM). LSTM is proposed to detect the human behavior based on Human Activity Recognition (HAR) dataset. The data is monitored and recorded with the aid of sensors like accelerometer and Gyroscope in the user smart phone. HAR dataset is collected from 30 persons, performing different activities with a smart phone to their waists. The testing of the model is evaluated with respect to accuracy and efficiency. The designed activity re...

International Journal of Engineering Applied Sciences and Technology

The study of wearable and handheld sensors recognizing human activity improved our understanding of human behaviours and human objectives. Many academics seek to identify the activities of a user from raw data using the fewest necessary resources. In this article, we propose a network of profound beliefs, a full-service architecture for the recognition of activities (DBN-LSTM). This DBN-LSTM method improves the human predictability of raw data and reduces the complexity of the model as well as the requirement for comprehensive workmanship. A geographically and temporally rich network is CNN-LSTM. Our proposed model for the UCI HAR Public Data Set can achieve 99% accuracy and 92% precision.

2021

This paper presents a novel approach to solve simultaneously the problems of human whole-body motion prediction and action recognition for real-time applications. Starting from the dynamics of human motion and motor system theory, the notion of mixture of experts from deep learning literature has been extended to solve this problem. The work is accompanied by 66-DoFs human model experiments.

2019

The problem of classifying body gesture and motion along with aiming to predict states of action or behaviour during physical activity is refered to as Human Activity Recongition (HAR). Inertial Measurement Units (IMUs) prevail as the key technique to measure range of motion, speed, velocity and magnetic field orientation during these physical activities. Onbody inertial sensors can be used to generate body motion and vital signs recording signals that can successfully learn models and accurately classify physical activities. In this paper, we compare the approaches of Extreme Gradient Boosting (XGBoost), Multilayer Perceptron (MLP), Convolutional Neural Network (CNN), Long Short-Term Memory Network (LSTM), CNN + LSTM Hybrid (ConvLSTM) and Autoencoder by Random Forest (AE w/ RF) to classify human activities on the MHEALTH dataset. All six of our classification models use raw, unstructured data obtained from 4 inertial onbody sensors. We examine multiple physical activities and on-bo...

Computer Vision and Image Understanding, 2021

Most of the existing works on human activity analysis focus on recognition or early recognition of the activity labels from complete or partial observations. Similarly, almost all of the existing video captioning approaches focus on the observed events in videos. Predicting the labels and the captions of future activities where no frames of the predicted activities have been observed is a challenging problem, with important applications that require anticipatory response. In this work, we propose a system that can infer the labels and the captions of a sequence of future activities. Our proposed network for label prediction of a future activity sequence has three branches where the first branch takes visual features from the objects present in the scene, the second branch takes observed sequential activity features, and the third branch captures the last observed activity features. The predicted labels and the observed scene context are then mapped to meaningful captions using a sequence-to-sequence learning-based method. Experiments on four challenging activity analysis datasets and a video description dataset demonstrate that our label prediction approach achieves comparable performance with the state-of-the-arts and our captioning framework outperform the state-of-the-arts.

International Conference on Computer Vision (ICCV), 2019

Inspired by human neurological structures for action anticipation , we present an action anticipation model that enables the prediction of plausible future actions by forecasting both the visual and temporal future. In contrast to current state-of-the-art methods which first learn a model to predict future video features and then perform action anticipation using these features, the proposed framework jointly learns to perform the two tasks, future visual and temporal representation synthesis, and early action anticipation. The joint learning framework ensures that the predicted future embeddings are informative to the action anticipation task. Furthermore, through extensive experimental evaluations we demonstrate the utility of using both visual and temporal semantics of the scene, and illustrate how this representation synthesis could be achieved through a recurrent Gen-erative Adversarial Network (GAN) framework. Our model outperforms the current state-of-the-art methods on multiple datasets: UCF101, UCF101-24, UT-Interaction and TV Human Interaction.

2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

In recent years, Deep Learning has been successfully applied to multimodal learning problems, with the aim of learning useful joint representations in data fusion applications. When the available modalities consist of time series data such as video, audio and sensor signals, it becomes imperative to consider their temporal structure during the fusion process. In this paper, we propose the Correlational Recurrent Neural Network (CorrRNN), a novel temporal fusion model for fusing multiple input modalities that are inherently temporal in nature. Key features of our proposed model include: (i) simultaneous learning of the joint representation and temporal dependencies between modalities, (ii) use of multiple loss terms in the objective function, including a maximum correlation loss term to enhance learning of cross-modal information, and (iii) the use of an attention model to dynamically adjust the contribution of different input modalities to the joint representation. We validate our model via experimentation on two different tasks: video-and sensor-based activity classification, and audiovisual speech recognition. We empirically analyze the contributions of different components of the proposed CorrRNN model, and demonstrate its robustness, effectiveness and state-of-the-art performance on multiple datasets.

Sensors, 2019

Recently, egocentric activity recognition has attracted considerable attention in the pattern recognition and artificial intelligence communities because of its wide applicability in medical care, smart homes, and security monitoring. In this study, we developed and implemented a deep-learning-based hierarchical fusion framework for the recognition of egocentric activities of daily living (ADLs) in a wearable hybrid sensor system comprising motion sensors and cameras. Long short-term memory (LSTM) and a convolutional neural network are used to perform egocentric ADL recognition based on motion sensor data and photo streaming in different layers, respectively. The motion sensor data are used solely for activity classification according to motion state, while the photo stream is used for further specific activity recognition in the motion state groups. Thus, both motion sensor data and photo stream work in their most suitable classification mode to significantly reduce the negative in...

arXiv (Cornell University), 2022

Human motion prediction is a complex task as it involves forecasting variables over time on a graph of connected sensors. This is especially true in the case of few-shot learning, where we strive to forecast motion sequences for previously unseen actions based on only a few examples. Despite this, almost all related approaches for few-shot motion prediction do not incorporate the underlying graph, while it is a common component in classical motion prediction. Furthermore, state-of-the-art methods for few-shot motion prediction are restricted to motion tasks with a fixed output space meaning these tasks are all limited to the same sensor graph. In this work, we propose to extend recent works on few-shot time-series forecasting with heterogeneous attributes with graph neural networks to introduce the first few-shot motion approach that explicitly incorporates the spatial graph while also generalizing across motion tasks with heterogeneous sensors. In our experiments on motion tasks with heterogeneous sensors, we demonstrate significant performance improvements with lifts from 10.4% up to 39.3% compared to best stateof-the-art models. Moreover, we show that our model can perform on par with the best approach so far when evaluating on tasks with a fixed output space while maintaining two magnitudes fewer parameters.

2009

The present paper describes the construction of a multimodal database, referred to as the PROMETHEUS database, which contains recordings from heterogeneous sensors. The main purpose of this database is the development of a framework for monitoring and interpretation of human behavior in unrestricted environments of both indoor and outdoor type. It contains single-person and multi-person scenarios, but also covers scenarios with interactions between groups of people. It is devoted to detection of typical and atypical events, while care has been to taken for the recordings to be as close to real-world conditions as possible. The uniqueness of the PROMETHEUS database comes not only from the unique sensor sets but is due primarily to its generic design, which allows for embracing a wide range of real-world applications (including smart-home and humanrobot interaction interfaces, indoors/outdoors public areas surveillance etc).