Transfer Entropy

Visual tracking of unknown objects in unconstrained video-sequences is extremely
challenging due to a number of unsolved issues. This thesis explores several of these
and examines possible approaches to tackle them.

The unconstrained nature of real-world input sequences creates huge variation in
the appearance of the target object due to changes in pose and lighting. Additionally,
the object can be occluded by either parts of itself, other elements of the scene, or
the frame boundaries. Observations may also be corrupted due to low resolution,
motion blur, large frame-to-frame displacement, or incorrect exposure or focus of the
camera. Finally, some objects are inherently difficult to track due to their (low) texture,
specular/transparent nature, non-rigid deformations, etc.

Conventional trackers depend heavily on the texture of the target. This causes issues
with transparent or untextured objects. Edge points can be used in cases where
standard feature points are scarce; these however suffer from the aperture problem. To
address this, the first contribution of this thesis explores the idea of virtual corners,
using pairs of non-adjacent line correspondences, tangent to edges in the image. Furthermore,
the chapter investigates the possibility of long-term tracking, introducing a
re-detection scheme to handle occlusions while limiting drift of the object model. The
outcome of this research is an edge-based tracker, able to track in scenarios including
untextured objects, full occlusions and significant length. The tracker, besides reporting
excellent results in standard benchmarks, is demonstrated to successfully track the
longest sequence published to date.

Some of the issues in visual tracking are caused by suboptimal utilisation of the
image information. The object of interest can easily occupy as few as ten or even one
percent of the video frame area. This causes difficulties in challenging scenarios such
as sudden camera shakes or full occlusions. To improve tracking in such cases, the next
major contribution of this thesis explores relationships within the context of visual
tracking, with a focus on causality. These include causal links between the tracked
object and other elements of the scene such as the camera motion or other objects.
Properties of such relationships are identified in a framework based on information
theory. The resulting technique can be employed as a causality-based motion model to
improve the results of virtually any tracker.

Significant effort has previously been devoted to rapid learning of object properties
on the fly. However, state-of-the-art approaches still often fail in cases such as
rapid out-of-plane rotations, when the appearance changes suddenly. One of the major
contributions of this thesis is a radical rethinking of the traditional wisdom of modelling
3D motion as appearance change. Instead, 3D motion is modelled as 3D motion.
This intuitive but previously unexplored approach provides new possibilities in visual
tracking research.

Firstly, 3D tracking is more general, as large out-of-plane motion is often fatal for 2D
trackers, but helps 3D trackers to build better models. Secondly, the tracker's internal
model of the object can be used in many different applications and it could even become
the main motivation, with tracking supporting reconstruction rather than vice versa.
This effectively bridges the gap between visual tracking and Structure from Motion.
The proposed method is capable of successfully tracking sequences with extreme out-of-plane
rotation, which poses a considerable challenge to 2D trackers. This is done by creating
realistic 3D models of the targets, which then aid in tracking.

In the majority of the thesis, the assumption is made that the target's 3D shape is
rigid. This is, however, a relatively strong limitation. In the final chapter, tracking and
dense modelling of non-rigid targets is explored, demonstrating results in even more
generic (and therefore challenging) scenarios. This final advancement truly generalises
the tracking problem with support for long-term tracking of low texture and non-rigid
objects in sequences with camera shake, shot cuts and significant rotation.

Taken together, these contributions address some of the major sources of failure
in visual tracking. The presented research advances the field of visual tracking, facilitating
tracking in scenarios which were previously infeasible. Excellent results are
demonstrated in these challenging scenarios. Finally, this thesis demonstrates that 3D
reconstruction and visual tracking can be used together to tackle difficult tasks.

We follow the main stocks belonging to the New York Stock Exchange and to Nasdaq from 2003 to 2012, through years of normality and of crisis, and study the dynamics of networks built on two measures expressing relations between those stocks: correlation, which is symmetric and measures how similar two stocks behave, and Transfer Entropy, which is non-symmetric and measures the influence of the time series of one stock onto another in terms of the information that the time series of one stock transmits to the time series of another stock. The two measures are used in the creation of two networks that evolve in time, revealing how the relations between stocks and between industrial sectors changed in times of crisis. The two networks are also used in conjunction with a dynamic model of the spreading of volatility in order to detect which are the stocks that are most likely to spread crises, according to the model. This information may be used in the building of policies aiming to reduce the effect of financial crises.

The Kullback–Leibler (KL) divergence is a fundamental measure of information geometry that is used in a variety of contexts in artificial intelligence. We show that, when system dynamics are given by distributed nonlinear systems, this measure can be decomposed as a function of two information-theoretic measures, transfer entropy and stochastic interaction. More specifically, these measures are applicable when selecting a candidate model for a distributed system, where individual subsystems are coupled via latent variables and observed through a filter. We represent this model as a directed acyclic graph (DAG) that characterises the unidirectional coupling between subsystems. Standard approaches to structure learning are not applicable in this framework due to the hidden variables; however, we can exploit the properties of certain dynamical systems to formulate exact methods based on differential topology. We approach the problem by using reconstruction theorems to derive an analytical expression for the KL divergence of a candidate DAG from the observed dataset. Using this result, we present a scoring function based on transfer entropy to be used as a subroutine in a structure learning algorithm. We then demonstrate its use in recovering the structure of coupled Lorenz and Rössler systems.

Finding interdependency relations between (possibly multivariate) time series provides valuable knowledge about the processes that generate the signals. Information theory sets a natural framework for non-parametric measures of several classes of statistical dependencies. However, a reliable estimation from information-theoretic functionals is hampered when the dependency to be assessed is brief or evolves in time. Here, we show that these limitations can be overcome when we have access to an ensemble of independent repetitions of the time series. In particular, we gear a dataefficient estimator of probability densities to make use of the full structure of trial-based measures. By doing so, we can obtain time-resolved estimates for a family of entropy combinations (including mutual information, transfer entropy, and their conditional counterparts) which are more accurate than the simple average of individual estimates over trials. We show with simulated and real data that the proposed approach allows to recover the time-resolved dynamics of the coupling between different subsystems.

We develop networks of international stock market indices using information and correlation based measures. We use 83 stock market indices of a diversity of countries, as well as their single day lagged values, to probe the correlation and the flow of information from one stock index to another taking into account different operating hours. Additionally, we apply the formalism of partial correlations to build the dependency network of the data, and calculate the partial Transfer Entropy to quantify the indirect influence that indices have on one another. We find that Transfer Entropy is an effective way to quantify the flow of information between indices, and that a high degree of information flow between indices lagged by one day coincides to same day correlation between them.

This dissertation consists of six distinct research studies that are broadly classified into two parts. The first part is concerned with the application of emerging data analysis tools rooted in causal inference, nonlinear chaotic dynamical systems, and information theory to detect associations and characterize patterns of interaction in complex hydrometeorological systems. This part is motivated by the rapid accumulation of hydrometeorological data records in the form of in-situ, remotely sensed observations and climatological reconstructions in addition to the significant advancements in data mining tools that facilitate discovery of interaction patterns solely from observational datasets. More specifically, I present four studies that utilize observational datasets to elucidate patterns of interaction and subsequently improve predictive understanding of the underlying processes. First, I evaluate the performance of four causal inference methods in recovering the causal structure underlying a hydrologic conceptual model and utilize causal analysis to formulate hypothesis on the differential impact of environmental variables in regulating evapotranspiration. Second, methods rooted in the theory of chaotic dynamical systems are used
to examine the dynamical properties of 400 hydrologic basins across the contiguous United States (CONUS) with the aim of developing a catchment classification framework. Third, I propose an algorithm that utilizes causal inference to extend methods of univariate state space forecasting to account for multivariate predictors. The algorithm is applied for daily streamflow forecasting in nine hydrologic basins across CONUS, and the results are compared to that of deep learning
models. Finally, concepts of information theory are used to diagnose the complex, nonlinear, space-time varying relationship between infrared brightness temperature and precipitation across different seasons and spatiotemporal scales.

The second part of the dissertation focuses on the use of long historical records of satellite-based precipitation datasets in hydroclimatic research, and it consists of two studies that utilize Precipitation Estimation from Remotely Sensed Information using Artificial Neural Networks –Climate Data Record (PERSIANN-CDR) dataset. The first study proposes a framework for developing Intensity Duration Frequency (IDF) curves from satellite-based precipitation datasets.
The framework accounts for the inherent biases in the estimates of PERSIANN-CDR, and it is used to develop IDF curves over CONUS with evaluation based on in-situ estimates of NOAA Atlas 14. The second study utilizes PERSIANN-CDR dataset over the Nile river basin to constrain future projections of precipitation obtained from climate models. More specifically, a Bayesian Model Averaging (BMA) approach is adopted to constrain future projections of 20 Global Climate Models (GCMs) from phase six of the Coupled Model Intercomparison Project (CMIP6). The results show that annual precipitation is projected to decrease in the upper White Nile basin, whereas projected change in the Blue Nile basin is highly uncertain both in magnitude and sign of
change.

Quantifying distributed information processing is crucial to understanding collective motion in animal groups. Recent studies have begun to apply rigorous methods based on information theory to quantify such distributed computation. Following this perspective, we use transfer entropy to quantify dynamic information flows locally in space and time across a school of fish during directional changes around a circular tank, i.e., U-turns. This analysis reveals peaks in information flows during collective U-turns and identifies two different flows: an informative flow (positive transfer entropy) from fish that have already turned to fish that are turning, and a misinformative flow (negative transfer entropy) from fish that have not turned yet to fish that are turning. We also reveal that the information flows are related to relative position and alignment between fish and identify spatial patterns of information and misinformation cascades. This study offers several methodological contributions and we expect further application of these methodologies to reveal intricacies of self-organisation in other animal groups and active matter in general.

Animals and humans engage in an enormous variety of behaviors which are orchestrated through a complex interaction of physical and informational processes: the physical interaction of the bodies with the environment is intimately coupled with informational processes in the animal's brain. A crucial step toward the mastery of all these behaviors seems to be to understand the flows of information in the sensorimotor networks. In this study, we have performed a quantitative analysis in an artificial agent -a running quadruped robot with multiple sensory modalities -using tools from information theory (transfer entropy). Starting from very little prior knowledge, through systematic variation of control signals and environment, we show how the agent can discover the structure of its sensorimotor space, identify proprioceptive and exteroceptive sensory modalities, and acquire a primitive body schema. In summary, we show how the analysis of directed information flows in an agent's sensorimotor networks can be used to bootstrap its perception and development.

This paper presents a data-driven pipeline for studying asymmetries in mutual interdependencies between distinct components of EEG signal. Due to volume conductance, estimating coherence between scalp electrodes may lead to spurious results. A group-based independent component analysis (ICA), which is conducted across all subjects and conditions simultaneously, is an alternative representation of the EEG measurements. Within this approach, the extracted components are independent in a global sense while short-lived or transient interdependencies may still be present between the components. In this paper, functional roles of the ICA components are specified through a partial least squares (PLS) analysis of task effects within the time course of the derived components. Functional integration is estimated within the information-theoretic approach using transfer entropy analysis based on asymmetries in mutual interdependencies of reconstructed phase dynamics. A secondary PLS analysis is performed to assess robust task-specific changes in transfer entropy estimates between functionally specific components.

The recurrent circuitry of the cerebral cortex generates an emergent pattern of activity that is organized into rhythmic periods of firing and silence referred to as slow oscillations (ca 1 Hz). Slow oscillations not only are dominant during slow wave sleep and deep anesthesia, but also can be generated by the isolated cortical network in vitro, being a sort of default activity of the cortical network. The cortex is densely and reciprocally connected with subcortical structures and, as a result, the slow oscillations in situ are the result of an interplay between cortex and thalamus. Due to this reciprocal connectivity and interplay, the mechanism responsible for the initiation of waves in the corticothalamocortical loop during slow oscillations is still a matter of debate. It was our objective to determine the directionality of the information flow between different layers of the cortex and the connected thalamus during spontaneous activity. With that purpose we obtained multilayer local field potentials from the rat visual cortex and from its connected thalamus, the lateral geniculate nucleus, during deep anaesthesia. We analyzed directionality of information flow between thalamus, cortical infragranular layers (5 and 6) and supragranular layers (2/3) by means of three information theoretical indicators: transfer entropy, symbolic transfer entropy and transcript mutual information. These three indicators coincided in finding that infragranular layers lead the information flow during slow oscillations both towards supragranular layers and towards the thalamus.

Causal relationships can often be found in visual object tracking between the motions of the camera and that of the tracked object. This object motion may be an effect of the camera motion, e.g. an unsteady handheld camera. But it may also be the cause, e.g. the cameraman framing the object. In this paper we explore these relationships, and provide statistical tools to detect and quantify them; these are based on transfer entropy and stem from information theory. The relationships are then exploited to make predictions about the object location. The approach is shown to be an excellent measure for describing such relationships. On the VOT2013 dataset the prediction accuracy is increased by 62% over the best non-causal predictor. We show that the location predictions are robust to camera shake and sudden motion, which is invaluable for any tracking algorithm and demonstrate this by applying causal prediction to two state-of-the-art trackers. Both of them benefit, Struck gaining a 7% accuracy and 22 % robustness increase on the VTB1.1 benchmark, becoming the new state-of-the-art.

ABSTRACT Early fault detection and isolation in industrial systems is vitally necessary to prevent any potential product damage. The paper proposes a new decentralized multi-unit fault isolation methodology in which all the known process faults with similar time signatures are grouped into appropriate categories. An innovative genetic algorithm-based method is introduced to explore for optimum plant zones in a large-scale plant wide search to appropriately configure each architectural unit, having less reliance on excess process variables with redundant and uncorrelated diagnostic information. The methodology employs a set of Bayes and radial basis function neural network classifiers to properly isolate the most usual known faults. A new idea based on transfer entropy algorithm has been integrated in the decentralized configuration to be triggered for isolation of novel faults which have been left unrecognized by the set of maintained classifiers. Experimental results clearly demonstrate that the proposed methods are considerably superior to the conventional centralized methods.

We develop networks of international stock market indices using information and correlation based measures. We use 83 stock market indices of a diversity of countries, as well as their single day lagged values, to probe the correlation and the flow of information from one stock index to another taking into account different operating hours. Additionally, we apply the formalism of partial correlations to build the dependency network of the data, and calculate the partial Transfer Entropy to quantify the indirect influence that indices have on one another. We find that Transfer Entropy is an effective way to quantify the flow of information between indices, and that a high degree of information flow between indices lagged by one day coincides to same day correlation between them.

Transfer entropy is a frequently employed measure of conditional co-dependence in non-parametric analysis of Granger causality. In this paper, we derive analytical expressions for transfer entropy for the multivariate exponential, logistic, Pareto (type I − IV) and Burr distributions. The latter two fall into the class of fat-tailed distributions with power law properties, used frequently in biological, physical and actuarial sciences. We discover that the transfer entropy expressions for all four distributions are identical and depend merely on the multivariate distribution parameter and the number of distribution dimensions. Moreover, we find that in all four cases the transfer entropies are given by the same decreasing function of distribution dimensionality.

1] Ecohydrological systems may be characterized as nonlinear, complex, open dissipative systems. Such systems consist of many coupled processes, and the couplings change depending on the system state or scale in space and time at which the system is analyzed. The arrangement of couplings in a complex system may be represented as a network of information flow and feedback between variables that measure system processes. The occurrence of feedback on such a network provides sufficient conditions for self-organized and nonlinear behaviors to emerge. We adapt an information-theoretic statistical method called transfer entropy for the purposes of robustly measuring the directionality, relative strength, statistical significance, and time scale of information flow between pairs of ecohydrological variables using time series data. A process network may be delineated where variables are cast as nodes and information flows as weighted directional links between them. The process network captures key couplings and time scales and represents the state of the complex system as a whole, including functional groups of variables (subsystems) and synchronization resulting from feedbacks. It is therefore able to identify interactions which are not detectable using methods which examine the system using one relationship at a time. We assemble an information flow process network using July 2003 FLUXNET data for a Midwestern corn-soybean ecohydrological system in a healthy, peak growing season state and compare the results with those using July 2005 data for the same site during a severe drought. We find that the process network during drought is substantially decoupled, and regional-scale information feedback is reduced during the drought. We conclude that the proposed process network methodology is able to identify the differences between two states of an ecohydrological system on the basis of variations in the pattern of feedback coupling on the network.

Transfer entropy (TE) is a recently proposed measure of the information flow between coupled linear or nonlinear systems. In this study, we suggest improvements in the selection of parameters for the estimation of TE that significantly enhance its accuracy and robustness in identifying the direction and the level of information flow between observed data series generated by coupled complex systems. We show the application of the improved TE method to long (in the order of days; approximately a total of 600 h across all patients), continuous, intracranial electroencephalograms (EEG) recorded in two different medical centers from four patients with focal temporal lobe epilepsy (TLE) for localization of their foci. All patients underwent ablative surgery of their clinically assessed foci. Based on a surrogate statistical analysis of the TE results, it is shown that the identified potential focal sites through the suggested analysis were in agreement with the clinically assessed sites of the epileptogenic focus in all patients analyzed. It is noteworthy that the analysis was conducted on the available whole-duration multielectrode EEG, that is, without any subjective prior selection of EEG segments or electrodes for analysis. The above, in conjunction with the use of surrogate data, make the results of this analysis robust. These findings suggest a critical role TE may play in epilepsy research in general, and as a tool for robust localization of the epileptogenic focus/foci in patients with focal epilepsy in particular.

This paper presents a data-driven pipeline for studying asymmetries in mutual interdependencies between distinct components of EEG signal. Due to volume conductance, estimating coherence between scalp electrodes may lead to spurious results. A group-based independent component analysis (ICA), which is conducted across all subjects and conditions simultaneously, is an alternative representation of the EEG measurements. Within this approach, the extracted components are independent in a global sense while short-lived or transient interdependencies may still be present between the components. In this paper, functional roles of the ICA components are specified through a partial least squares (PLS) analysis of task effects within the time course of the derived components. Functional integration is estimated within the information-theoretic approach using transfer entropy analysis based on asymmetries in mutual interdependencies of reconstructed phase dynamics. A secondary PLS analysis is performed to assess robust task-specific changes in transfer entropy estimates between functionally specific components.