Human Action Recognition

Human Action Recognition has seen significant advances through transformer-based architectures, yet achieving a nuanced understanding often requires fusing multiple data modalities. Standard models relying solely on RGB video can struggle with actions defined by subtle motion cues rather than appearance. This paper introduces TransMODAL, a novel dual-stream transformer that synergistically fuses spatiotemporal appearance features from a pre-trained VideoMAE(Video Masked AutoEncoders) backbone with explicit skeletal kinematics from a state-of-the-art pose estimation pipeline (RT-DETR(Real-Time DEtection Transformer) + ViTPose++). We propose two key architectural innovations to enable effective and efficient fusion: a CoAttentionFusion module that facilitates deep, iterative cross-modal feature exchange between the RGB and pose streams, and an efficient AdaptiveSelector mechanism that dynamically prunes less informative spatiotemporal tokens to reduce computational overhead. Evaluated on three challenging benchmarks, TransMODAL demonstrates robust generalization, achieving accuracies of 98.5% on KTH, 96.9% on UCF101, and 84.2% on HMDB51. These results significantly outperform a strong VideoMAE-only baseline and are competitive with state-of-the-art methods, demonstrating the profound impact of explicit pose guidance. TransMODAL presents a powerful and efficient paradigm for composing pre-trained foundation models to tackle complex video understanding tasks by providing a fully reproducible implementation and strong benchmark results.

Human action recognition is a topic widely studied over time, using numerous techniques and methods to solve a fundamental problem in automatic video analysis. Basically, a traditional human action recognition system collects video frames of human activities, extracts the desired features of each human skeleton and classify them to distinguish human gesture. However, almost all of these approaches roll out the space-time information of the recognition process. In this paper we present a novel use of an existing state-of-the-art space-time technique, the Space-Time Interest Point (STIP) detector and its velocity adaptation, to human action recognition process. Using STIPs as descriptors and a Support Vector Machine classifier, we evaluate four different public video datasets to validate our methodology and demonstrate its accuracy in real scenarios.

The Silberstein model of the molecular polarizability of diatomic molecules, generalized by Applequist et al. for polyatomic molecules, is analyzed. The atoms are regarded as isotropically polarizable points located at their nuclei, interacting via the fields of their induced dipoles. The use of additive values for atom polarizabilities gives poor results, in some cases leading to artificial predictions of absorption bands. The molecular polarizability of methane and its derivative are computed. The agreement with experimental mean molecular polarizabilities is within 1-5%. A hypothesis is indispensable for a suitable representation of polarizability derivative.

Most state-of-the-art action localization systems process each action proposal individually, without explicitly exploiting their relations during learning. However, the relations between proposals actually play an important role in action localization, since a meaningful action always consists of multiple proposals in a video. In this paper, we propose to exploit the proposal-proposal relations using Graph Convolutional Networks (GCNs). First, we construct an action proposal graph, where each proposal is represented as a node and their relations between two proposals as an edge. Here, we use two types of relations, one for capturing the context information for each proposal and the other one for characterizing the correlations between distinct actions. Then we apply the GCNs over the graph to model the relations among different proposals and learn powerful representations for the action classification and localization. Experimental results show that our approach significantly outperforms the state-of-the-art on THUMOS14 (49.1% versus 42.8%). Moreover, augmentation experiments on ActivityNet also verify the efficacy of modeling action proposal relationships. Codes are available at .

Most state-of-the-art action localization systems process each action proposal individually, without explicitly exploiting their relations during learning. However, the relations between proposals actually play an important role in action localization, since a meaningful action always consists of multiple proposals in a video. In this paper, we propose to exploit the proposal-proposal relations using Graph Convolutional Networks (GCNs). First, we construct an action proposal graph, where each proposal is represented as a node and their relations between two proposals as an edge. Here, we use two types of relations, one for capturing the context information for each proposal and the other one for characterizing the correlations between distinct actions. Then we apply the GCNs over the graph to model the relations among different proposals and learn powerful representations for the action classification and localization. Experimental results show that our approach significantly outperforms the state-of-the-art on THUMOS14 (49.1% versus 42.8%). Moreover, augmentation experiments on ActivityNet also verify the efficacy of modeling action proposal relationships. Codes are available at .

Collaborative-based representation classifiers have widely spread in the latest years achieving remarkable results in signal and image processing tasks. In this paper, we consider these approaches for the hyperspectral image classification. Specifically, we focus on collaborative and sparse representation classifiers and we perform an investigation on the role of the different regularizations and constraints that can be considered with respect to the classification performance. In addition, we propose to consider the Nearest Subspace Classifier with regularization which, from the experiments, has proven to be a competitive classification technique. Experimental results have been conducted considering both spectral and spatial features of a real hyperspectral image.

A method to monitor elderly people in an indoor environment using conventional cameras is presented. The method can be used to identify people's activities and initiate suitable actions as needed. The originality of our approach is in combining spatial and temporal contexts with the position and orientation for the detected person. Preliminary evaluation, based only on the first two features (spatial and temporal), achieved the accuracy over 60% in a realistic residential environment. Although the results are based on using only two out of the four proposed input features, they already demonstrate a promising improvement over using a single feature in isolation.

Unintentional falls are a major public health concern for many communities, especially with aging populations. There are various approaches used to classify human activities for fall detection. Related studies have employed wearable, non-invasive sensors, video cameras and depth sensor-based approaches to develop such monitoring systems. The proposed approach in this study uses a depth sensor and employs a unique procedure which identifies the fall risk levels to adapt the algorithm for different people with their physical strength to withstand falls. The inclusion of the fall risk level identification, further enhanced and improved the accuracy of the fall detection. The experimental results showed promising performance in adapting the algorithm for people with different fall risk levels for fall detection.

Current state-of-the-art approaches for spatio-temporal action detection deal with stable videos and quite sterilized environments, as seen in the UCF-101 benchmark. In addition, the objects of interest are typically relatively close to the camera, and therefore fairly clear and easily distinguished. This study presents an approach method for online human action detection in long-distance imaging affected by atmospheric distortions. We created a unique dataset of typical actions in long-range imaging. Various CNN frameworks were examined for the initial moving object detection phase, including 2D, 3D, one stream, and two-stream (RGB frames and optical flow). The basic object detection methods examined within these frameworks include the YOLOv3 and an extension of the inflated 3D ConvNet with a Feature-Fused Single Shot Multibox Detector (FFSSD) to improve small object detection. To cope with the harmful effect of the spatio-temporal random movements induced by atmospheric effects on motion estimation, we first fit the optical flow stream characteristics to a temporally noisy turbulent environment. A significant improvement of the action detection quality under such noisy conditions was obtained by constructing an online tracking algorithm that incrementally constructs and labels the objects' tracks from the network's frame-level detections. Experimental results show that our approach outperforms the state-of-the-art on our dataset in terms of the mAP measure.

This paper presents a survey of trajectory-based activity analysis for visual surveillance. It describes techniques that use trajectory data to define a general set of activities that are applicable to a wide range of scenes and environments. Events of interest are detected by building a generic topographical scene description from underlying motion structure as observed over time. The scene topology is automatically learned and is distinguished by points of interest and motion characterized by activity paths. The methods we review are intended for real-time surveillance through definition of a diverse set of events for further analysis triggering, including virtual fencing, speed profiling, behavior classification, anomaly detection, and object interaction.

In this paper we address the task of recognizing assembly actions as a structure (e.g. a piece of furniture or a toy block tower) is built up from a set of primitive objects. Recognizing the full range of assembly actions requires perception at a level of spatial detail that has not been attempted in the action recognition literature to date. We extend the fine-grained activity recognition setting to address the task of assembly action recognition in its full generality by unifying assembly actions and kinematic structures within a single framework. We use this framework to develop a general method for recognizing assembly actions from observation sequences, along with observation features that take advantage of a spatial assembly's special structure. Finally, we evaluate our method empirically on two application-driven data sources: (1) An IKEA furniture-assembly dataset, and (2) A blockbuilding dataset. On the first, our system recognizes assembly actions with an average framewise accuracy of 70% and an average normalized edit distance of 10%. On the second, which requires fine-grained geometric reasoning to distinguish between assemblies, our system attains an average normalized edit distance of 23%-a relative improvement of 69% over prior work.

In this paper we address the task of recognizing assembly actions as a structure (e.g. a piece of furniture or a toy block tower) is built up from a set of primitive objects. Recognizing the full range of assembly actions requires perception at a level of spatial detail that has not been attempted in the action recognition literature to date. We extend the fine-grained activity recognition setting to address the task of assembly action recognition in its full generality by unifying assembly actions and kinematic structures within a single framework. We use this framework to develop a general method for recognizing assembly actions from observation sequences, along with observation features that take advantage of a spatial assembly's special structure. Finally, we evaluate our method empirically on two application-driven data sources: (1) An IKEA furniture-assembly dataset, and (2) A blockbuilding dataset. On the first, our system recognizes assembly actions with an average framewise accuracy of 70% and an average normalized edit distance of 10%. On the second, which requires fine-grained geometric reasoning to distinguish between assemblies, our system attains an average normalized edit distance of 23%-a relative improvement of 69% over prior work.

In this report, the technical details of our submission to the EPIC-Kitchens Action Anticipation Challenge 2021 are given. We developed a hierarchical attention model for action anticipation, which leverages Transformer-based attention mechanism to aggregate features across temporal dimension, modalities, symbiotic branches respectively. In terms of Mean Top-5 Recall of action, our submission with team name ICL-SJTU achieved 13.39% for overall testing set, 10.05% for unseen subsets and 11.88% for tailed subsets. Additionally, it is noteworthy that our submission ranked 1st in terms of verb class in all three (sub)sets.

Recognizing human actions through video analysis has gained significant attention in applications like surveillance, sports analytics, and human-computer interaction. While deep learning models such as 3D convolutional neural networks (CNNs) and recurrent neural networks (RNNs) deliver promising results, they often struggle with computational inefficiencies and inadequate spatial-temporal feature extraction, hindering scalability to larger datasets or high-resolution videos. To address these limitations, we propose a novel model combining a two-dimensional convolutional restricted Boltzmann machine (2D Conv-RBM) with a long short-term memory (LSTM) network. The 2D Conv-RBM efficiently extracts spatial features such as edges, textures, and motion patterns while preserving spatial relationships and reducing parameters via weight sharing. These features are subsequently processed by the LSTM to capture temporal dependencies across frames, enabling effective recognition of both short-and long-term action patterns. Additionally, a smart frame selection mechanism minimizes frame redundancy, significantly lowering computational costs without compromising accuracy. Evaluation on the KTH, UCF Sports, and HMDB51 datasets demonstrated superior performance, achieving accuracies of 97.3%, 94.8%, and 81.5%, respectively. Compared to traditional approaches like 2D RBM and 3D CNN, our method offers notable improvements in both accuracy and computational efficiency, presenting a scalable solution for real-time applications in surveillance, video security, and sports analytics.

In 3D human motion pose-based analysis, the main problem is how to classify multi-class label activities based on primitive action (pose) inputs efficiently for both accuracy and processing time. Because, pose is not unique and the same pose can be anywhere on different activity classes. In this paper, we evaluate the effectiveness of Extreme Learning Machine (ELM) in 3D human motion analysis based on pose cluster. ELM has reputation as eager classifier with fast training and testing time but the classification result originally has still low testing accuracy even by increasing the hidden nodes number and adding more training data. To achieve better accuracy, we pursue a feature selection method to reduce the dimension of pose cluster training data in time sequence. We propose to use frequency of pose occurrence. This method is similar like bag of words which is a sparse vector of occurrence counts of poses in histogram as features for training data (bag of poses). By using bag of poses as the optimum feature selection, the ELM performance can be improved without adding network complexity (Hidden nodes number and training data).

This paper presents a human action recognition method using dynamic time warping and voting algorithms on 3D human skeletal models. In this method human actions, which are the combinations of multiple body part movements, are described by feature matrices concerning both spatial and temporal domains. The feature matrices are created based on the spatial selection of relative angles between body parts in time series. Then, action recognition is done by applying a classifier which is the combination of dynamic time warping (DTW) and a voting algorithm to the feature matrices. Experimental results show that the performance of our action recognition method obtains high recognition accuracy at reliable computation speed and can be applied in real time human action recognition systems.

The rapid development of autonomous
vehicles (AVs) has led to significant
advancements in scene understanding, a
critical aspect of ensuring safe and efficient
navigation. Deep learning (DL) has emerged
as a powerful tool for processing and
interpreting complex sensory data from
various onboard sensors such as cameras,
LiDAR, and radar. This paper explores the
role of deep learning techniques in enhancing
scene understanding for autonomous vehicles,
encompassing object detection, semantic
segmentation, depth estimation, and behavior
prediction. We delve into state-of-the-art deep
neural networks, including convolutional
neural networks (CNNs), recurrent neural
networks (RNNs), and transformer-based
models, highlighting their applications in
detecting and classifying dynamic and static
objects, such as pedestrians, vehicles, traffic
signs, and road infrastructure. Additionally,
we examine advancements in multi-modal
fusion, which combines data from different
sensors to improve accuracy and robustness in
diverse driving conditions. The abstract also
addresses challenges such as domain
adaptation, real-time processing, and the need
for large-scale annotated datasets. Solutions
leveraging transfer learning, synthetic data
generation, and active learning are discussed
to overcome these limitations.

The increasing complexity of decisionmaking in critical domains such as healthcare, finance, and emergency response has highlighted the need for a collaborative approach that integrates human intuition and expertise with artificial intelligence (AI)'s analytical power. This paper presents a comprehensive framework for human-AI collaboration in complex decision-making scenarios. The framework is designed to leverage the complementary strengths of humans and AI by fostering seamless interaction, mutual understanding, and adaptive workflows. Our approach addresses key challenges, including trust building,

The rapid advancement of Artificial Intelligence (AI) is revolutionizing online education by offering new opportunities for creating personalized learning pathways tailored to individual learners. This paper explores how AI-powered systems are transforming the landscape of online education by adapting content, pacing, and assessments to meet diverse learner needs. Personalized learning pathways leverage AI algorithms to analyze learner data, such as performance metrics, engagement patterns, and preferences, to curate customized educational experiences. Key applications of AI include intelligent tutoring systems,

Video-based action recognition is a
crucial task in computer vision with
applications spanning surveillance, sports
analytics, human-computer interaction, and
autonomous systems. This paper explores the
application of spatiotemporal deep learning
models for action recognition, leveraging
advancements in neural network architectures
to analyze video data effectively. Traditional
approaches often process spatial and temporal
dimensions independently, limiting their
ability to capture complex motion patterns
and contextual relationships. In contrast,
spatiotemporal deep learning models
integrate spatial and temporal features
simultaneously, enabling robust recognition of
dynamic actions. The study highlights key
methods, including convolutional neural
networks (CNNs) for spatial feature extraction
and recurrent neural networks (RNNs) or 3D
convolutional networks (3D-CNNs) for
temporal modelling. It also delves into
transformer-based architectures and attention
mechanisms, which enhance model
performance by selectively focusing on salient
regions and time steps.

The Low-light photography often suffers from issues such as noise, low contrast, and color distortion, which degrade image quality and limit its applicability in various fields such as night-time photography, security surveillance, and autonomous driving. Traditional image enhancement techniques struggle to adequately address these challenges due to the complex nature of noise and illumination variations in low-light conditions. In recent years, deep learning models have emerged as powerful tools for enhancing low-light images, offering significant improvements over conventional methods. This paper reviews the state-of-the

Human Action Recognition is a challenging issue in the real time constraints where the action videos or images are contaminated with several side effects like noises, moving backgrounds, multiple views, hindered movements etc. Under these constraints, to recognize an action, we have developed a new Human Action Recognition system. Under this system, an edge efficient action descriptor called as Laplacian Histogram of Gradients is proposed through which the all possible movements of an action are extracted. Further, to ensure a perfect discrimination between different action descriptors, we have employed kernel discriminant analysis. The proposed recognition model is evaluated systematically on a standard action dataset, IXMAS. Experimental results prove that our method outperforms the existing methods in terms of recognition accuracy.

Correct classication of image data can depend on features learned in multiple sequential frames. We focus on the problem of learning action from video data with an emphasis on driver behavior monitoring. An insucient quantity of high quality labeled data is a major problem in machine learning research. is is especially true when deep neural networks are used. Although some suciently large, general purpose image databases exist for action recognition, most of these are limited to single frames. is kind of data requires that the action recognition task is applied regardless of the temporal information (information from previous and next frames of a video sequence). In this paper, we show that temporal information is useful for accurate classication of video and that the temporal information in lower layers of a convolutional neural network can successfully be transferred from one network to another to greatly improve performance on the driver behavior monitoring task.

This paper presents a human action recognition method using dynamic time warping and voting algorithms on 3D human skeletal models. In this method human actions, which are the combinations of multiple body part movements, are described by feature matrices concerning both spatial and temporal domains. The feature matrices are created based on the spatial selection of relative angles between body parts in time series. Then, action recognition is done by applying a classifier which is the combination of dynamic time warping (DTW) and a voting algorithm to the feature matrices. Experimental results show that the performance of our action recognition method obtains high recognition accuracy at reliable computation speed and can be applied in real time human action recognition systems.

Human Activity Recognition (HAR) is a crucial component of computer vision, with applications in human-computer interaction and surveillance. As the need for HAR technology keeps increasing, so does the desire for solutions that can help people train by showcasing professional moves. For instance, new recruits can be successfully trained in particular fighting skills by observing the activities of seasoned soldiers. In order to increase the accuracy and dependability of HAR systems, this study investigates the incorporation of human motion dynamics into Convolutional Neural Network (CNN) architectures. This study enhances CNN's ability to capture both spatial and temporal features by incorporating dynamic changes in human movement as additional inputs, which results in a more complex comprehension of human activity. A significant identification of complex human activity and frequent movement is made viable by the architecture's proficiency in uniting motion data with classic graphic information. Experimentations operated on prominent datasets reveal that motion dynamics significantly enhance recognition exactness, mainly under challenging circumstances like occlusions, inconsistent viewpoints, and complicated actions. This study highlights how motion-informed CNN architectures can enhance HAR classification and open new avenues for multimodal action recognition research.

Computing optical flow is a fundamental problem in computer vision. However, deep learning-based optical flow techniques do not perform well for non-rigid movements such as those found in faces, primarily due to lack of the training data representing the fine facial motion. We hypothesize that learning optical flow on face motion data will improve the quality of predicted flow on faces. This work aims to: (1) exploring self-supervised techniques to generate optical flow ground truth for face images; (2) computing baseline results on the effects of using face data to train Convolutional Neural Networks (CNN) for predicting optical flow; and (3) using the learned optical flow in micro-expression recognition to demonstrate its effectiveness. We generate optical flow ground truth using facial key-points in the BP4D-Spontaneous dataset. This optical flow is used to train the FlowNetS architecture to test its performance on the Extended Cohn-Kanade dataset and a portion of the generated dataset. The performance of FlowNetS trained on face images surpassed that of other optical flow CNN architectures. Our optical flow features are further compared with other methods using the STSTNet micro-expression classifier, and the results indicate that the optical flow obtained using this work has promising applications in facial expression analysis.

Human  Action  Recognition is  used  to  analyse  the  videos  to  identify  the  actions  performed  by humans.  In  recent  years,  it  has  gained  much  popularity  due  to  its  large  domain  of  applications  presented  in various  fields.  Several  research  contributions  are  available  in  this  area  but  still  the  requirement  is  to  achieve good  results  for  various  challenging  datasets  and  limited  hardware  resources.    In  order  toovercome  these issues,  an  ensemble  deep  learning  model  is  proposed  in  this  paper  based  on custom convolutional  neural network  (CNN). ResNet50  is  merged  with  a  handcrafted  CNN,  to  identify  human  actions in challenging video datasets. This  model is  trained  and  tested  on UCF-101  and  HMDB-51  datasets  and  gained  very  good  results. The experimental results showcased that the proposed model outperforms some recent works in this domain.

Due to the rising demand for automatic interpretation of visual relationships in several domains, human-object interaction (HOI) detection and recognition have also gained more attention from researchers over the last decade. This survey paper concentrates on human-centric interactions, which can be categorized as human-to-human and human-to-objects. Although an extensive amount of research work has been done in this area, real-world constraints like the domain of possible interactions make the research a challenging task. This paper provides an analysis of conventional hand-crafted representation-based methods and recent deep learning-based methods, ongoing advancements taking place in the field of HOI recognition and detection, and challenges faced by the researchers. Moreover, we present a detailed picture of publicly available datasets for HOI evaluations. At the end, the future scope of the study is discussed.

Spatial and temporal relationships, both short-range and long-range, between objects in videos, are key cues for recognizing actions. It is a challenging problem to model them jointly. In this paper, we first present a new variant of Long Short-Term Memory, namely Relational LSTM, to address the challenge of relation reasoning across space and time between objects. In our Relational LSTM module, we utilize a non-local operation similar in spirit to the recently proposed non-local network to substitute the fully connected operation in the vanilla LSTM. By doing this, our Relational LSTM is capable of capturing long and short-range spatio-temporal relations between objects in videos in a principled way. Then, we propose a two-branch neural architecture consisting of the Relational LSTM module as the non-local branch and a spatio-temporal pooling based local branch. The local branch is utilized for capturing local spatial appearance and/or short-term motion features. The two branches a...

Stacked denoising autoencoders (SDAs) have been successfully used to learn new representations for domain adaptation. Recently, they have attained record accuracy on standard benchmark tasks of sentiment analysis across different text domains. SDAs learn robust data representations by reconstruction, recovering original features from data that are artificially corrupted with noise. In this paper, we propose marginalized SDA (mSDA) that addresses two crucial limitations of SDAs: high computational cost and lack of scalability to high-dimensional features. In contrast to SDAs, our approach of mSDA marginalizes noise and thus does not require stochastic gradient descent or other optimization algorithms to learn parameters -in fact, they are computed in closed-form. Consequently, mSDA, which can be implemented in only 20 lines of MATLAB TM , significantly speeds up SDAs by two orders of magnitude. Furthermore, the representations learnt by mSDA are as effective as the traditional SDAs, attaining almost identical accuracies in benchmark tasks.

This paper examines the connection between Ludwig von Mises and early contributors to game theory. What becomes clear is that early game theorists were trained by Austrians who thus influenced the field from its beginning. The connection most commonly known is the influence Mises had on Oskar Morgenstern, however, this paper reveals a previously unknown connection between Mises, Bert Hozelitz, and John Nash.

Activity detection in security videos is a difficult problem due to multiple factors such as large field of view, presence of multiple activities, varying scales and viewpoints, and its untrimmed nature. The existing research in activity detection is mainly focused on datasets, such as UCF-101, JHMDB, THUMOS, and AVA, which partially address these issues. The requirement of processing the security videos in realtime makes this even more challenging. In this work we propose Gabriella, a real-time online system to perform activity detection on untrimmed security videos. The proposed method consists of three stages: tubelet extraction, activity classification, and online tubelet merging. For tubelet extraction, we propose a localization network which takes a video clip as input and spatio-temporally detects potential foreground regions at multiple scales to generate action tubelets. We propose a novel Patch-Dice loss to handle large variations in actor size. Our online processing of videos at a clip level drastically reduces the computation time in detecting activities. The detected tubelets are assigned activity class scores by the classification network and merged together using our proposed Tubelet-Merge Action-Split (TMAS) algorithm to form the final action detections. The TMAS algorithm efficiently connects the tubelets in an online fashion to generate action detections which are robust against varying length activities. We perform our experiments on the VIRAT and MEVA (Multiview Extended Video with Activities) datasets and demonstrate the effectiveness of the proposed approach in terms of speed (∼100 fps) and performance with state-of-the-art results. The code and models will be made publicly available.

Activity detection in surveillance videos is a difficult problem due to multiple factors such as large field of view, presence of multiple activities, varying scales and viewpoints, and its untrimmed nature. The existing research in activity detection is mainly focused on datasets, such as UCF-101, JHMDB, THUMOS, and AVA, which partially address these issues. The requirement of processing the surveillance videos in real-time makes this even more challenging. In this work we propose Gabriella, a real-time online system to perform activity detection on untrimmed surveillance videos. The proposed method consists of three stages: tubelet extraction, activity classification, and online tubelet merging. For tubelet extraction, we propose a localization network which takes a video clip as input and spatio-temporally detects potential foreground regions at multiple scales to generate action tubelets. We propose a novel Patch-Dice loss to handle large variations in actor size. Our online processing of videos at a clip level drastically reduces the computation time in detecting activities. The detected tubelets are assigned activity class scores by the classification network and merged together using our proposed Tubelet-Merge Action-Split (TMAS) algorithm to form the final action detections. The TMAS algorithm efficiently connects the tubelets in an online fashion to generate action detections which are robust against varying length activities. We perform our experiments on the VIRAT and MEVA (Multiview Extended Video with Activities) datasets and demonstrate the effectiveness of the proposed approach in terms of speed (~100 fps) and performance with state-of-the-art results. The code and models will be made publicly available.

In this paper, we describe a simple strategy for mitigating variability in temporal data series by shifting focus onto long-term, frequency domain features that are less susceptible to variability. We apply this method to the human action recognition task and demonstrate how working in the frequency domain can yield good recognition features for commonly used optical flow and articulated pose features, which are highly sensitive to small differences in motion, viewpoint, dynamic backgrounds, occlusion and other sources of variability. We show how these frequency-based features can be used in combination with a simple forest classifier to achieve good and robust results on the popular KTH Actions dataset. * This work was partially supported by the DARPA Minds Eye program. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of DARPA or the U.S. Government.

In this paper, we describe a simple strategy for mitigating variability in temporal data series by shifting focus onto long-term, frequency domain features that are less susceptible to variability. We apply this method to the human action recognition task and demonstrate how working in the frequency domain can yield good recognition features for commonly used optical flow and articulated pose features, which are highly sensitive to small differences in motion, viewpoint, dynamic backgrounds, occlusion and other sources of variability. We show how these frequency-based features can be used in combination with a simple forest classifier to achieve good and robust results on the popular KTH Actions dataset. * This work was partially supported by the DARPA Minds Eye program. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of DARPA or the U.S. Government.

Human Action Recognition (HAR) has gained increasing importance in various fields, including
surveillance, healthcare, and sports analysis. This paper presents the development of a Human
Action Recognition model using the Long-term Recurrent Convolutional Network (LCRN)
architecture, which combines Convolutional Neural Networks (CNNs) for spatial feature
extraction and Long Short-Term Memory (LSTM) for temporal sequence modeling. The model
processes video frames to recognize actions such as walking, running, and jumping. The LCRN
model was trained and tested on a benchmark action recognition dataset, achieving an accuracy
of 87.5%. The model's ability to capture both spatial and temporal information from video
sequences makes it suitable for real-time applications requiring action recognition. Our
experiments show that the LCRN architecture outperforms models that only utilize CNNs or
RNNs by effectively modeling the dynamics of human movement over time

The severity of sustained injury resulting from assault-related violence can be minimised by reducing detection time. However, it has been shown that human operators perform poorly at detecting events found in video footage when presented with simultaneous feeds. We utilise computer vision techniques to develop an automated method of abnormal crowd detection that can aid a human operator in the detection of violent behaviour. We observed that behaviour in city centre environments often occur in crowded areas, resulting in individual actions being occluded by other crowd members. We propose a real-time descriptor that models crowd dynamics by encoding changes in crowd texture using temporal summaries of Grey Level Co-Occurrence Matrix (GLCM) features. We introduce a measure of inter-frame uniformity (IFU) and demonstrate that the appearance of violent behaviour changes in a less uniform manner when compared to other types of crowd behaviour. Our proposed method is computationally cheap and offers real-time description. Evaluating our method using a privately held CCTV dataset and the publicly available Violent Flows, UCF Web Abnormality, and UMN Abnormal Crowd datasets, we report a receiver operating characteristic score of 0.9782, 0.9403, 0.8218 and 0.9956 respectively.

Vision-based Human Action Recognition (HAR) is a hot topic in computer vision. Recently, deep-based HAR has shown promising results. HAR using a single data modality is a common approach; however, the fusion of different data sources essentially conveys complementary information and improves the results. This paper comprehensively reviews deep-based HAR methods using multiple visual data modalities. The main contribution of this paper is categorizing existing methods into four levels, which provides an in-depth and comparable analysis of approaches in various aspects. So, at the first level, proposed methods are categorized based on the employed modalities. At the second level, methods categorized in the first level are classified based on the employment of complete modalities or working with missing modalities at the test time. At the third level, complete and missing modality branches are categorized based on existing approaches. Finally, similar frameworks in the third category are grouped together. In addition, a comprehensive comparison is provided for publicly available benchmark datasets, which helps to compare and choose suitable datasets for a task or to develop new datasets. This paper also compares the performance of state-of-the-art methods on benchmark datasets. The review concludes by highlighting several future directions.

This paper describes our method developed for TRECVID 2012 Semantic INdexing (SIN) Task. Our main research purpose is the development of a fast method, which can work on a single processor with no performance degradation. To this end, computationally expensive processes are re-formulated based on matrix operation. We re-formulate the Euclidian distance computation for the kernel value computation in an SVM, and the probability density computation of multivariate normal distributions for the GMM supervector representation. This enables accurate concept detection using a large number of training examples, and spatially-temporally dense features. The following four runs were submitted to SIN (light) task: • L A kobe muro l5 4: This is our baseline run using five features, 1. SIFT at Harris-Affine regions, 2. SIFT at Hessian-Affine regions, 3. Trajectory displacement, 4. HOG around trajectories, and 5. MFCC. Five SVMs built on these features are fused using the weighted linear combination approach. This run achieved the MAP 0.320. • L A kobe muro l6 1: In addition to the above five features, this run uses the sixth feature, Spatio-Temporal-Dense RGB SIFT (STD-RGB-SIFT), consisting of SIFT descriptors sampled at every sixth pixel in every other frame. The extraction of this feature becomes feasible because of the significant speedup of the probability density computation. This run achieved the MAP 0.348. • L A kobe muro l18 3: To cover the diversity of a concept's appearances, this run utilizes bagging where three SVMs are built on each of six features using different subsets of training examples. Such many SVMs can be built due to the fast kernel value computation. SVMs are fused using the weighted linear combination. This run achieved the MAP 0.358, which is the highest score among all the runs submitted to SIN (light) task. • L A kobe muro r18 2: This run uses rough set theory to fuse SVMs in L A kobe muro l18 3, and achieved the MAP 0.323. The above results indicate the effectiveness of the spatiallytemporally dense feature STD-RGB-SIFT. In particular, the MAP 0.302 was achieved only using STD-RGB-SIFT. This is significantly higher than MAPs using the other single features. Also, the effectiveness of bagging can be seen from the above results.

This paper aims to determine which is the best human action recognition method based on features extracted from RGB-D devices, such as the Microsoft Kinect. A review of all the papers that make reference to MSR Action3D, the most used dataset that includes depth information acquired from a RGB-D device, has been performed. We found that the validation method used by each work differs from the others. So, a direct comparison among works cannot be made. However, almost all the works present their results comparing them without taking into account this issue. Therefore, we present different rankings according to the methodology used for the validation in orden to clarify the existing confusion.

Since the Microsoft Kinect has been released, the usage of marker-less body pose estimation has been enormously eased. Based on 3D skeletal pose information, complex human gestures and actions can be recognised in real time. However, due to errors in tracking or occlusions, the obtained information can be noisy. Since the RGB-D data is available, the 3D or 2D shape of the person can be used instead. However, depending on the viewpoint and the action to recognise, it might present a low discriminative value. In this paper, the combination of body pose estimation and 2D shape, in order to provide additional characteristic value, is considered so as to improve human action recognition. Using efficient feature extraction techniques, skeletal and silhouette-based features are obtained which are low dimensional and can be obtained in real time. These two features are then combined by means of feature fusion. The proposed approach is validated using a stateof-the-art learning method and the MSR Action3D dataset as benchmark. The obtained results show that the fused feature achieves to improve the recognition rates, outperforming state-of-the-art results in recognition rate and robustness.

Due to progress and demographic change, society is facing a crucial challenge related to increased life expectancy and a higher number of people in situations of dependency. As a consequence, there exists a significant demand for support systems for personal autonomy. This article outlines the vision@home project, whose goal is to extend independent living at home for elderly and impaired people, providing care and safety services by means of vision-based monitoring. Different kinds of ambient-assisted living services are supported, from the detection of home accidents, to telecare services. In this contribution, the specification of the system is presented, and novel contributions are made regarding human behaviour analysis and privacy protection. By means of a multi-view setup of cameras, people's behaviour is recognised based on human action recognition. For this purpose, a weighted feature fusion scheme is proposed to learn from multiple views. In order to protect the right to privacy of the inhabitants when a remote connection occurs, a privacy-by-context method is proposed. The experimental results of the behaviour recognition method show an outstanding performance, as well as support for multi-view scenarios and real-time execution, which are required in order to provide the proposed services.

Interest in RGB-D devices is increasing due to their low price and the wide range of possible applications that come along. These devices provide a marker-less body pose estimation by means of skeletal data consisting of 3D positions of body joints. These can be further used for pose, gesture or action recognition. In this work, an evolutionary algorithm is used to determine the optimal subset of skeleton joints, taking into account the topological structure of the skeleton, in order to improve the final success rate. The proposed method has been validated using a state-of-the-art RGB action recognition approach, and applying it to the MSR-Action3D dataset. Results show that the proposed algorithm is able to significantly improve the initial recognition rate and to yield similar or better success rates than the state-of-the-art methods.

We propose a self-supervised visual learning method by predicting the variable playback speeds of a video. Without semantic labels, we learn the spatio-temporal visual representation of the video by leveraging the variations in the visual appearance according to different playback speeds under the assumption of temporal coherence. To learn the spatio-temporal visual variations in the entire video, we have not only predicted a single playback speed but also generated clips of various playback speeds and directions with randomized starting points. Hence the visual representation can be successfully learned from the meta information (playback speeds and directions) of the video. We also propose a new layerdependable temporal group normalization method that can be applied to 3D convolutional networks to improve the representation learning performance where we divide the temporal features into several groups and normalize each one using the different corresponding parameters. We validate the effectiveness of our method by fine-tuning it to the action recognition and video retrieval tasks on UCF-101 and HMDB-51. a

This paper describes a novel methodology for automated recognition of high-level activities. A key aspect of our framework relies on the concept of cooccurring visual words for describing interactions between several persons. Motivated by the numerous success of human activity recognition methods using bag-of-words, this paradigm is extended. A 3-D XYT spatio-temporal volume is generated for each interacting person and a set of visual words is extracted to represent his activity. The interaction is then represented by the frequency of co-occurring visual words between persons. For our experiments, we used the UT-interaction dataset which contains several complex human-human interactions.

Due to the growing use of human action recognition in every day life applications, it has become one of the very hot topics in image analysis and pattern recognition. This paper presents a new feature extraction method for human action recognition. The method is based on the extraction of Trace transforms from binarized silhouettes, representing different stages of a single action period. A final history template com-posed from the above transforms, represents the whole se-quence containing much of the valuable spatio-temporal in-formation contained in a human action. The new method takes advantage of the natural specifications of the specific Trace transform, such as noise robustness, translation invari-ance and scalability easiness and produces effective, simple and fast created features. Classification experiments performed on KTH action database using Radial Basis Function (RBF) Kernel SVM, provided very competitive results indicating the potentials of the proposed technique.

Image processing is a method that takes image as input and performs some operation on it and gives image as output. It takes input as photograph or video frame and gives output as the parameters related to that image. Image processing will enhance the image by taking the useful information from it .Image processing will take the image as two dimensional signals. Two methods are used for image processing analog and digital image processing. Image processing is a growing technology and can be used in a various fields like optical imaging, medical imaging, computer vision, computer graphics etc. This paper gives a broad view of various digital image processing techniques.

In modern e-Healthcare systems, human activity recognition (HAR) is one of the most challenging tasks in remote monitoring of patients suffering from mental illness or disabilities for necessary assistance. One of the major issues is to provide security to a number of different connected devices to the Internet, known as Internet of Things (IoT). A potential solution to this problem is the blockchain-based architecture. In addition, the complex nature of activities performed by humans in diverse healthcare environments reduces the qualitative measures for extracting distinct features representing various human actions. To answer this challenge, we propose an activity monitoring and recognition framework, which is based on multi-class cooperative categorization procedure to improve the activity classification accuracy in videos supporting the fog or cloud computing-based blockchain architecture. In the proposed approach, frame-based salient features are extracted from videos consisting of different human activities, which are further processed into action vocabulary for efficiency and accuracy. Similarly, the classifica-

Digital-enabled manufacturing systems require a high level of automation for fast and low-cost production but should also present flexibility and adaptiveness to varying and dynamic conditions in their environment, including the presence of human beings; however, this presence of workers in the shared workspace with robots decreases the productivity, as the robot is not aware about the human position and intention, which leads to concerns about human safety. This issue is addressed in this work by designing a reliable safety monitoring system for collaborative robots (cobots). The main idea here is to significantly enhance safety using a combination of recognition of human actions using visual perception and at the same time interpreting physical human–robot contact by tactile perception. Two datasets containing contact and vision data are collected by using different volunteers. The action recognition system classifies human actions using the skeleton representation of the latter w...