Event Modeling and Recognition Using Markov Logic Networks

ACM Transactions on Computational Logic, 2015

Symbolic event recognition systems have been successfully applied to a variety of application domains, extracting useful information in the form of events, allowing experts or other systems to monitor and respond when significant events are recognised. In a typical event recognition application, however, these systems often have to deal with a significant amount of uncertainty. In this paper, we address the issue of uncertainty in logic-based event recognition by extending the Event Calculus with probabilistic reasoning. Markov Logic Networks are a natural candidate for our logic-based formalism. However, the temporal semantics of the Event Calculus introduce a number of challenges for the proposed model. We show how and under what assumptions we can overcome these problems. Additionally, we study how probabilistic modelling changes the behaviour of the formalism, affecting its key property, the inertia of fluents. Furthermore, we demonstrate the advantages of the probabilistic Event Calculus through examples and experiments in the domain of activity recognition, using a publicly available dataset for video surveillance.

Proceedings of the 15th ACM on International conference on multimodal interaction - ICMI '13, 2013

We present a general framework for complex event recognition that is well-suited for integrating information that varies widely in detail and granularity. Consider the scenario of an agent in an instrumented space performing a complex task while describing what he is doing in a natural manner. The system takes in a variety of information, including objects and gestures recognized by RGB-D and descriptions of events extracted from recognized and parsed speech. The system outputs a complete reconstruction of the agent's plan, explaining actions in terms of more complex activities and filling in unobserved but necessary events. We show how to use Markov Logic (a probabilistic extension of first-order logic) to create a model in which observations can be partial, noisy, and refer to future or temporally ambiguous events; complex events are composed from simpler events in a manner that exposes their structure for inference and learning; and uncertainty is handled in a sound probabilistic manner. We demonstrate the effectiveness of the approach for tracking kitchen activities in the presence of noisy and incomplete observations.

Journal of Artificial Intelligence Research

Event models obtained automatically from video can be used in applications ranging from abnormal event detection to content based video retrieval. When multiple agents are involved in the events, characterizing events naturally suggests encoding interactions as relations. Learning event models from this kind of relational spatio-temporal data using relational learning techniques such as Inductive Logic Programming (ILP) hold promise, but have not been successfully applied to very large datasets which result from video data. In this paper, we present a novel framework REMIND (Relational Event Model INDuction) for supervised relational learning of event models from large video datasets using ILP. Efficiency is achieved through the learning from interpretations setting and using a typing system that exploits the type hierarchy of objects in a domain. The use of types also helps prevent over generalization. Furthermore, we also present a type-refining operator and prove that it is optim...

Computer Vision and Image Understanding, 2016

This paper presents a new framework for multi-subject event inference in surveillance video, where measurements produced by low-level vision analytics usually are noisy, incomplete or incorrect. Our goal is to infer the composite events undertaken by each subject from noise observations. To achieve this, we consider the temporal characteristics of event relations and propose a method to correctly associate the detected events with individual subjects. The Dempster-Shafer (DS) theory of belief functions is used to infer events of interest from the results of our vision analytics and to measure conflicts occurring during the event association. Our system is evaluated against a number of videos that present passenger behaviours on a public transport platform namely buses at different levels of complexity. The experimental results demonstrate that by reasoning with spatio-temporal correlations, the proposed method achieves a satisfying performance when associating atomic events and recognising composite events involving multiple subjects in dynamic environments.

IET Computer Vision, 2008

The authors propose a high-level scenario recognition algorithm for video sequence interpretation. The recognition of scenarios is based on a Bayesian networks approach. The model of a scenario contains two main layers. The first one allows events from the observed visual features to be highlighted and the second layer is focused on the temporal reasoning stage. The temporal layer uses specific nodes permitting an event-based approach. These nodes focus on the lifetime of events highlighted from the results of the first layer. The temporal layer then estimates the qualitative and quantitative relations between the different temporal events helpful for the recognition task. The global recognition algorithm is illustrated over real indoor image sequences of an abandoned baggage scenario.

IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 2000

Understanding Video Events, the translation of low-level content in video sequences into highlevel semantic concepts, is a research topic that has received much interest in recent years. Important applications of this work include smart surveillance systems, semantic video database indexing, and interactive systems. This technology can be applied to several video domains including: airport terminal, parking lot, traffic, subway stations, aerial surveillance, and sign language data. In this work we survey the two main components of the event understanding process: Abstraction and Event modeling. Abstraction is the process of molding the data into informative units to be used as input to the event model. Event modeling is devoted to describing events of interest formally and enabling recognition of these events as they occur in the video sequence. Event modeling can be further decomposed in the categories of Pattern Recognition Methods, State Event Models, and Semantic Event Models. In this survey we discuss this proposed taxonomy of the literature, offer a unifying terminology, and discuss popular abstraction schemes (e.g. Motion History Images) and event modeling formalisms (e.g. Hidden Markov Model) and their use in video event understanding using extensive examples from the literature. Finally we consider the application domain of video event understanding in light of the proposed taxonomy, and propose future directions for research in this field. 1

Understanding complex, dynamic scenes of real-world activities from low-level sensor data is of central importance for intelligent systems. The main difficulty lies in the fact that complex scenes are best described in high-level, logical formalisms, while sensor data usually consists of many low-level features. We first propose a method to obtain a logical representation of real-world, dynamic scenes based on input video stream solely. We focus on representing the video data using probabilistic relational sequences as a natural way to incorporate sensor uncertainty. They allow us to work with structured terms, and in addition they capture the inherent uncertainty of object detection. Further on, we employ r-grams as the probabilistic logical learning model for this application. In a first step we use r-grams in a simple setting and we show their viability in card games. We also show how r-grams can be upgraded to deal with uncertain observations.

2007

Video surveillance is the process of monitoring the behavior of people and objects within public places, e.g. airports and traffic intersections, by means of visual aids (cameras) usually for safety and security purposes. As the amount of video data gathered daily by surveillance cameras increases, the need for automatic systems to detect and recognize suspicious activities performed by people and objects is also increasing. The first part of the thesis describes a framework for modeling and recognition of events from surveillance video. Our framework is based on deterministic inference using Petri nets. Events can be composed by combining primitive events and previously defined events by spatial, temporal and logical relations. We provide a graphical user interface (GUI) to formulate such event models. Our approach automatically maps each of these models into a set of Petri net filters that represent the components of the event. Last but definitely not least, I would like to thank my two lovely daughters; Farah and Sarah for their unconditional love and patience during completing this thesis. iv TABLE OF CONTENTS List of Figures ix above and to the left of (x; y), inclusive. (b) The sum of the pixels within rectangle D is computed as: ii(4) + ii(1) − ii(2) − ii(3). .. .. .. . 4.4 Examples from Backpack Dataset Showing Different Ways of Holding the Backpack .

2008

Behavior understanding and semantic interpretation of dynamic visual scenes have attracted a lot of attention in computer vision research community. Although the use of surveillance cameras has proliferated, the understanding of activities still remains complex. While users are mostly interested in high level and subjective semantics, only low level visual features can be extracted in a reliable way. This paper presents a novel framework for video guided behavior monitoring, built around the event modeling concept. It enables users to design their personal models of events combining elementary concept and low level features using expressive formalisms. The framework enables then detection of the events within video streams based on low level features extraction and manual annotations analysis, while taking in consideration uncertainty. Examples depicting content-based events modeling and detection from video surveillance are presented to illustrate the approach.

Proceedings of the 4th ACM International Conference on Distributed Event-Based Systems, DEBS 2010, 2010

Today's organisations require techniques for automated transformation of the large data volumes they collect during their operations into operational knowledge. This requirement may be addressed by employing event recognition systems that detect activities/events of special significance within an organisation, given streams of 'low-level' information that is very difficult to be utilised by humans. Numerous event recognition systems have been proposed in the literature. Recognition systems with a logic-based representation of event structures, in particular, have been attracting considerable attention because, among others, they exhibit a formal, declarative semantics, they haven proven to be efficient and scalable, and they are supported by machine learning tools automating the construction and refinement of event structures. In this paper we review representative approaches of logic-based event recognition, and discuss open research issues of this field.