Transfer Entropy

Many multivariate time series anomaly detection frameworks have been proposed and widely applied. However, most of these frameworks do not consider intrinsic relationships between variables in multivariate time series data, thus ignoring the causal relationship among variables and degrading anomaly detection performance. This work proposes a novel framework called CGAD, an entropy Causal Graph for multivariate time series Anomaly Detection. CGAD utilizes transfer entropy to construct graph structures that unveil the underlying causal relationships among time series data. Weighted graph convolutional networks combined with causal convolutions are employed to model both the causal graph structures and the temporal patterns within multivariate time series data. Furthermore, CGAD applies anomaly scoring, leveraging median absolute deviation-based normalization to improve the robustness of the anomaly identification process. Extensive experiments demonstrate that CGAD outperforms state-of-the-art methods on real-world datasets with a 9% average improvement in terms of three different multivariate time series anomaly detection metrics. CCS Concepts: • Computing methodologies → Anomaly detection; Causal reasoning and diagnostics; Neural networks.

Background/Introduction: Widespread network disruption has been hypothesized to be an important predictor of outcomes in patients with refractory temporal lobe epilepsy (TLE). Most studies examining functional network disruption in epilepsy have largely focused on the symmetric bidirectional metrics of the strength of network connections. However, a more complete description of network dysfunction impacts in epilepsy requires an investigation of the potentially more sensitive directional metrics of information flow. Methods: This study describes a whole-brain magnetoencephalography-imaging approach to examine restingstate directional information flow networks, quantified by phase-transfer entropy (PTE), in patients with TLE compared with healthy controls (HCs). Associations between PTE and clinical characteristics of epilepsy syndrome are also investigated. Results: Deficits of information flow were specific to alpha-band frequencies. In alpha band, while HCs exhibit a clear posterior-to-anterior directionality of information flow, in patients with TLE, this pattern of regional information outflow and inflow was significantly altered in the frontal and occipital regions. The changes in information flow within the alpha band in selected brain regions were correlated with interictal spike frequency and duration of epilepsy. Conclusions: Impaired information flow is an important dimension of network dysfunction associated with the pathophysiological mechanisms of TLE.

The functional brain activity network is the result of the harmonious activity of different regions. Each harmonic activity requires feedback from different areas of activity to adjust itself. As a result, any disorder in this harmony can affect the functional brain activity network. In this study, we use transfer entropy (TE) to examine the information transfer patterns of brain activity in Healthy Controls (HC) and individuals with Autism Spectrum Disorder (ASD). The results indicate that the pattern of information transfer between brain canonical networks is different in these two groups, ASDs and HCs. The results of our study demonstrate that the HC group exhibits a higher volume of feedback in comparison to the ASD group. In addition, the major sources of information flow in HCs are the Default Mode Network and Visual Network, but the hubs in the ASD group are the Frontoparietal Network and Limbic. Also, the backbone extracted from the TE graph of brain regions of interest shows us the same modularity class in the flow of information in both groups, but we observe the separated fragments in ASDs. Finally, the question that can be answered is whether the pattern of information transfer can be used as a biomarker in diagnosing ASDs vs. HCs. To explore this, we employed XGBoost, which achieved a mean classification accuracy of 91% in distinguishing individuals with ASDs from HCs across cross-validations.

Transfer entropy (TE) captures the directed relationships between two variables. Partial transfer entropy (PTE) accounts for the presence of all confounding variables of a multivariate system and infers only about direct causality. However, the computation of partial transfer entropy involves high dimensional distributions and thus may not be robust in case of many variables. In this work, different variants of the partial transfer entropy are introduced, by building a reduced number of confounding variables based on different scenarios in terms of their interrelationships with the driving or response variable. Connectivity-based PTE variants utilizing the random forests (RF) methodology are evaluated on synthetic time series. The empirical findings indicate the superiority of the suggested variants over transfer entropy and partial transfer entropy, especially in the case of high dimensional systems. The above findings are further highlighted when applying the causality measures on...

Information causality measures have proven to be very effective in uncovering the connectivity patterns of multivariate systems. The non-uniform embedding (NUE) scheme has been developed to address the “curse of dimensionality”, since the estimation relies on high-dimensional conditional mutual information (CMI) terms. Although the NUE scheme is a dimension reduction technique, the estimation of high-dimensional CMIs is still required. A possible solution is the utilization of low-dimensional approximation (LA) methods for the computation of CMIs. In this study, we aim to provide useful insights regarding the effectiveness of causality measures that rely on NUE and/or on LA methods. In a comparative study, three causality detection methods are evaluated, namely partial transfer entropy (PTE) defined using uniform embedding, PTE using the NUE scheme (PTENUE), and PTE utilizing both NUE and an LA method (LATE). Results from simulations on well known coupled systems suggest the superio...

Information causality measures have proven to be very effective in uncovering the connectivity patterns of multivariate systems. The non-uniform embedding (NUE) scheme has been developed to address the "curse of dimensionality", since the estimation relies on high-dimensional conditional mutual information (CMI) terms. Although the NUE scheme is a dimension reduction technique, the estimation of high-dimensional CMIs is still required. A possible solution is the utilization of low-dimensional approximation (LA) methods for the computation of CMIs. In this study, we aim to provide useful insights regarding the effectiveness of causality measures that rely on NUE and/or on LA methods. In a comparative study, three causality detection methods are evaluated, namely partial transfer entropy (PTE) defined using uniform embedding, PTE using the NUE scheme (PTENUE), and PTE utilizing both NUE and an LA method (LATE). Results from simulations on well known coupled systems suggest the superiority of PTENUE over the other two measures in identifying the true causal effects, having also the least computational cost. The effectiveness of PTENUE is also demonstrated in a real application, where insights are presented regarding the leading forces in financial data.

The entropy production in stochastic dynamical systems is linked to the structure of their causal representation in terms of Bayesian networks. Such a connection was formalized for bipartite (or multipartite) systems with an integral fluctuation theorem in [Phys. Rev. Lett. 111, 180603 (2013)]. Here we introduce the information thermodynamics for time series, that are non-bipartite in general, and we show that the link between irreversibility and information can only result from an incomplete causal representation. In particular, we consider a backward transfer entropy lower bound to the conditional time series irreversibility that is induced by the absence of feedback in signal-response models. We study such a relation in a linear signal-response model providing analytical solutions, and in a nonlinear biological model of receptor-ligand systems where the time series irreversibility measures the signaling efficiency.

The irreversibility of trajectories in stochastic dynamical systems is linked to the structure of their causal representation in terms of Bayesian networks. We consider stochastic maps resulting from a time discretization with interval τ of signal-response models, and we find an integral fluctuation theorem that sets the backward transfer entropy as a lower bound to the conditional entropy production. We apply this to a linear signal-response model providing analytical solutions, and to a nonlinear model of receptor-ligand systems. We show that the observational time τ has to be fine-tuned for an efficient detection of the irreversibility in time-series.

Recent theoretical and empirical work has focused on the variability of network dynamics in maturation. Such variability seems to reflect the spontaneous formation and dissolution of different functional networks. We sought to extend these observations into healthy aging. Two different data sets, one EEG (total n = 48, ages 18-72) and one magnetoencephalography (n = 31, ages 20-75) were analyzed for such spatiotemporal dependency using multiscale entropy (MSE) from regional brain sources. In both data sets, the changes in MSE were timescale dependent, with higher entropy at fine scales and lower at more coarse scales with greater age. The signals were parsed further into local entropy, related to information processed within a regional source, and distributed entropy (information shared between two sources, i.e., functional connectivity). Local entropy increased for most regions, whereas the dominant change in distributed entropy was age-related reductions across hemispheres. These data further the understanding of changes in brain signal variability across the lifespan, suggesting an inverted U-shaped curve, but with an important qualifier. Unlike earlier in maturation, where the changes are more widespread, changes in adulthood show strong spatiotemporal dependence.

Identification of causal structures and quantification of direct information flows in complex systems is a challenging yet important task, with practical applications in many fields. Data generated by dynamical processes or large-scale systems are often symbolized, either because of the finite resolution of the measurement apparatus, or because of the need of statistical estimation. By algorithmic application of causation entropy, we investigated the effects of symbolization on important concepts such as Markov order and causal structure of the tent map. We uncovered that these quantities depend nonmontonically and, most of all, sensitively on the choice of symbolization. Indeed, we show that Markov order and causal structure do not necessarily converge to their original analog counterparts as the resolution of the partitioning becomes finer.

Synchronization of chaotic oscillators has become well characterized by errors which shrink relative to a synchronization manifold. This manifold is the identity function in the case of identical systems, or some other slow manifold in the case of generalized synchronizaton in the case of nonidentical components. On the other hand, since many decades beginning with the Smale horseshoe, chaotic oscillators can be well understood in terms of symbolic dynamics as information producing processes. We study here the synchronization of a pair of chaotic oscillators as a process for sharing information bearing bits transferred between each other, by measuring the transfer entropy tracked as the global system transitions to the synchronization state. Further, we present for the first time the notion of transfer entropy in the measure theoretic setting of transfer operators.

A basic systems question concerns the concept of closure, meaning autonomy (closed) in the sense of describing the (sub)system as fully consistent within itself. Alternatively, the system may be nonautonomous (open), meaning it receives influence from an outside subsystem. We assert here that the concept of information flow and the related concept of causation inference are summarized by this simple question of closure as we define herein. We take the forecasting perspective of Weiner-Granger causality that describes a causal relationship exists if a subsystem's forecast quality depends on considering states of another subsystem. Here, we develop a new direct analytic discussion, rather than a data oriented approach. That is, we refer to the underlying Frobenius-Perron (FP) transfer operator that moderates evolution of densities of ensembles of orbits, and two alternative forms of the restricted Frobenius-Perron operator, interpreted as if either closed (deterministic FP) or not...

Causal inference is perhaps one of the most fundamental concepts in science, beginning originally from the works of some of the ancient philosophers, through today, but also weaved strongly in current work from statisticians, machine learning experts, and scientists from many other fields. This paper takes the perspective of information flow, which includes the Nobel prize winning work on Granger-causality, and the recently highly popular transfer entropy, these being probabilistic in nature. Our main contribution will be to develop analysis tools that will allow a geometric interpretation of information flow as a causal inference indicated by positive transfer entropy. We will describe the effective dimensionality of an underlying manifold as projected into the outcome space that summarizes information flow. Therefore, contrasting the probabilistic and geometric perspectives, we will introduce a new measure of causal inference based on the fractal correlation dimension conditionall...

We propose an entropy statistic designed to assess the behavior of slowly varying parameters of real systems. Based on correlation entropy, the method uses symbol dynamics and analysis of increments to achieve sufficient recurrence in a short time series to enable entropy measurements on small data sets. We analyze entropy along a moving window of a time series, the entropy statistic tracking the behavior of slow variables of the data series. We employ the technique against several physiological time series to illustrate its utility in characterizing the constraints on a physiological time series. We propose that changes in the entropy of measured physiological signal (e.g. power output) during dynamic exercise will indicate changes in underlying constraint of the system of interest. This is compelling because CE may serve as a non-invasive, objective means of determining physiological stress under non-steady state conditions such as competition or acute clinical pathologies. If so, CE could serve as a valuable tool for dynamically monitoring health status in a wide range of non-stationary systems.

Synchronization of chaotic oscillators has become well characterized by errors which shrink relative to a synchronization manifold. This manifold is the identity function in the case of identical systems, or some other slow manifold in the case of generalized synchronizaton in the case of nonidentical components. On the other hand, since many decades beginning with the Smale horseshoe, chaotic oscillators can be well understood in terms of symbolic dynamics as information producing processes. We study here the synchronization of a pair of chaotic oscillators as a process for sharing information bearing bits transferred between each other, by measuring the transfer entropy tracked as the global system transitions to the synchronization state. Further, we present for the first time the notion of transfer entropy in the measure theoretic setting of transfer operators.

This work proposes a unified thermodynamic and semantic theory of civilizational collapse, termed Entropology—the study of entropy as it manifests not only in energy flows but also in symbolic, legal, and infrastructural systems.

Building upon systems theory, ecological thermodynamics, semiotics, and collapse science, the volume argues that civilizations do not fall due to resource scarcity alone, but because their ability to redirect entropy—through law, language, technology, and spatial externalization—eventually fails.

Core concepts include:

Entropy Transfer Chains (ETC)

Semantic Overload

Collapse Gradient

Thermodynamic Cities

The Great Filter as Entropic Singularity

Rather than predicting doom, the book offers a framework to understand why meaning collapses before matter, and how symbolic infrastructures (laws, myths, interfaces) become brittle long before buildings fall.

It closes by asking: Can symbolic mutation outpace entropy accumulation?
If so, what new attractors lie beyond empire?

This is Volume I of a multi-part series exploring collapse, survival, and post-collapse meaning systems.

The analogy between information theory and income distribution analysis is exploited to derive a number of measures of distributional change. These include not only counterparts of the regular entropy measures, but also of the entire. rich 'generalised entropy' family. The paper analyses the properties of these measures, and in particular their performance in response to LindependenV'transfers and to cases where an income transfer in the Id distribution automatically produces a corresponding transfer in the new distribution. The relationship of such 'functionally dependent* transfers to a measure of tax progression is examined. 'Although Thcil &~ckvcd the cowept in terms of inequality measurement. its origin ma!' br traced back to Shannan (194s). 'Here and elsewhere we shall assume for convenience that this ratio is formed \1> dl~iding the larger income by the smaller.

This article presents a structured literature review of dynamic volatility spillovers between spot and futures markets during financial crises, applying the TCCM (Theory, Context, Characteristics, Methodology) framework. Drawing from influential and Scopus-indexed studies, it offers a consolidated understanding of volatility dynamics, key transmission mechanisms, and methodological advancements. The study also identifies evolving trends, critical research gaps, and future research directions relevant to academics and financial practitioners.

Trajectories from a pair of interacting zebrafish are used to test for the existence of anticipatory dynamics in natural systems. Anticipatory dynamics (AD) is unusual in that causal events are not necessarily ordered by their temporal order. However, their causal order can still be established if the direction of information flow (DIF) is known. In order to obtain DIF between trajectories of the two fish, we have made use of the difference of the transfer entropy between the trajectories with a history length established by experiments with known DIF. Our experimental results indicate that AD can be observed much more often in fish pairs of different genders. The use of DIF to determine causal order is further verified by the simulation of two chaotic Lorenz oscillators with anticipatory coupling; mimicking the interaction between the fish. Our simulation results further suggest that the two fish are interacting with their own internal dynamics, not by adaptation.

Measures of complexity are of immediate interest for the field of autonomous robots both as a means to classify the behavior and as an objective function for the autonomous development of robot behavior. In the present paper we consider predictive information in sensor space as a measure for the behavioral complexity of a chain of two-wheel robots which are passively coupled and controlled by a closed-loop reactive controller for each of the individual robots. The predictive information, the mutual information between the past and the future of a time series, is approximated by restricting the time horizons to a single time step. This is exact for Markovian systems but seems to work well also for our robotic system which is strongly non-Markovian.When in a maze with many obstacles, the approximated predictive information of the sensor values of an individual robot is found to have a clear maximum for a controller which realizes the spontaneous cooperation of the robots in the chain so that large areas of the maze can be visited.

Human brain activity maps are produced by functional MRI (fMRI) research that describes the average level of engagement during a specific task of various brain regions. Functional connectivity describes the interrelationship, integrated performance, and organization of these different brain regions. This study investigates functional connectivity to quantify the interactions between different brain regions engaged concurrently in a specific task. The key focus of this study was to introduce and demonstrate task-specific functional connectivity among brain regions using fMRI data and decode cognitive states by proposing a novel classifier using connectivity features. Two connectivity models were considered: a graph-based taskspecific functional connectivity and a Granger causality-transfer entropy framework. Connectivity strengths obtained among brain regions were used for cognitive state classification. The parameters of the nodal and global graph analysis from the graph-based connectivity framework were considered, and the transfer entropy values of the causal connectivity model were considered as features for the cognitive state classification. The proposed model achieved an average accuracy of 95% on the StarPlus fMRI dataset and showed an improvement of 5% compared to the existing Tensor-SVD classification algorithm.

In a recent work we proposed the corrected transfer entropy (CTE), which reduces the bias in the estimation of transfer entropy (TE), a measure of Granger causality for bivariate time series making use of the conditional mutual information. An extension of TE to account for the presence of other time series is the partial TE (PTE). Here, we propose the correction of PTE, termed Corrected PTE (CPTE), in a similar way to CTE: time shifted surrogates are used in order to quantify and correct the bias, and the estimation of the involved entropies of high-dimensional variables is made with the method of k-nearest neighbors. CPTE is evaluated on coupled stochastic systems with both linear and nonlinear interactions. Finally, we apply CPTE to economic data and investigate whether we can detect the direct causal effects among economic variables.

In the aim to explore the complex relationships between S&P500, VIX and volume we introduce a Granger causality test using the nonlinear statistic of Asymmetric Partial Transfer Entropy (APTE). Through a simulation exercise, it arises that the APTE offers precise information on the nature of the connectivity. Our empirical findings concretize the information flow that links volume, S&P500 and VIX, and merge the leverage effect and the asymmetric stock return-volume relationship into a unified framework of analysis. More specifically, when we condition on the tails, the detected causal channel provides empirical validation of the noise trading contribution to large swings in financial markets, because of the increase of trading volume and the subsequent worsening ability of market prices to adjust to new information.

In this paper, a framework is developed for the identification of causal effects from nonstationary time series. Focusing on causality measures that make use of delay vectors from time series, the idea is to account for non-stationarity by considering the ranks of the components of the delay vectors rather than the components themselves. As an exemplary measure, we introduce the partial symbolic transfer entropy (PSTE), which is an extension of the bivariate symbolic transfer entropy (STE) quantifying only the direct causal effects among the variables of a multivariate system. Through Monte Carlo simulations it is shown that the PSTE is directly applicable to non-stationary in mean and variance time series and it is not affected by the existence of outliers and VAR filtering. For stationary time series, the PSTE is also compared to the linear conditional Granger causality index (CGCI). Finally, the causal effects among three financial variables are investigated. Computations of the PSTE and the CGCI on both the initial returns and the VAR filtered returns, and the PSTE on the original non-stationary time series, show consistency of the PSTE in estimating the causal effects.

Measures of the direction and strength of the interdependence among time series from multivariate systems are evaluated based on their statistical significance and discrimination ability. The best-known measures estimating direct causal effects, both linear and nonlinear, are considered, i.e., conditional Granger causality index (CGCI), partial Granger causality index (PGCI), partial directed coherence (PDC), partial transfer entropy (PTE), partial symbolic transfer entropy (PSTE) and partial mutual information on mixed embedding (PMIME). The performance of the multivariate coupling measures is assessed on stochastic and chaotic simulated uncoupled and coupled dynamical systems for different settings of embedding dimension and time series length. The CGCI, PGCI and PDC seem to outperform the other causality measures in the case of the linearly coupled systems, while the PGCI is the most effective one when latent and exogenous variables are present. The PMIME outweighs all others in the case of nonlinear simulation systems.

Information causality measures, i.e. transfer entropy and symbolic transfer entropy, are modified using the concept of surrogate data in order to identify correctly the presence and direction of causal effects. The measures are evaluated on multiple bivariate time series of known coupled systems of varying complexity and on a range of embedding dimensions. The proposed modifications of the causality measures are found to reduce the bias in the estimation of the measures and preserve the zero level in the absence of coupling.

An extension of transfer entropy, called partial transfer entropy, is proposed here to detect causal effects among observed interacting systems, and particularly distinguish among direct and indirect causal effects. The measure is presented for time series from three interacting systems, but can be extended for more time series. Partial transfer entropy is evaluated on simulated and real data and is compared with partial directed coherence, a linear causality measure defined in the frequency domain. The simulation study showed that both measures can identify linear and nonlinear causal effects for the correct parameters. Both measures are then tested on electroencephalograms of epileptic patients for the detection of the direction of the interaction among brain areas during the preictal, ictal and postictal states.

In this paper, we introduce the partial symbolic transfer entropy (PSTE), an extension of the symbolic transfer entropy that accounts only for the direct causal effects among the components of a multivariate system. It is an information theoretic measure, and as such does not suffer from model mis-specification bias. The PSTE is defined on the ranks of vectors that are formed from the reconstructed vectors, instead of the original time series values. The statistical significance of * Corresponding author 1 PSTE is assessed by randomization test making use of surrogate time series. The PSTE is evaluated on multivariate time series of different types of coupled and uncoupled systems and compared with conditional Granger causality index (CGCI). It is shown that the PSTE is not affected by the existence of outliers, it is directly applicable to time series that are non-stationary in mean and in variance, and it is also not affected by data filtering. As a real application, the causal effects among three economic indexes are investigated. Computations of PSTE and CGCI on both the initial returns and the VAR filtered returns, and only of PSTE on the original indexes, showed consistency of the PSTE in estimating the causal effect.

An extension of transfer entropy, called partial transfer entropy (PTE), is proposed to detect causal effects among observed interacting systems, and particularly to distinguish direct from indirect causal effects. PTE is compared to a linear direct causality measure, the Partial Directed Coherence (PDC), on known linear stochastic systems and nonlinear deterministic systems. PTE performs equally well as PDC on the linear systems and better than PDC on the nonlinear systems, both being dependent on the selection of the measure specific parameters. PTE and PDC are applied to electroencephalograms of epileptic patients during the preictal, ictal and postictal states, and PTE turns out to detect better changes of the strength of the direct causality at specific pairs of electrodes and for the different states.

Measures of the direction and strength of the interdependence between two time series are evaluated and modified in order to reduce the bias in the estimation of the measures, so that they give zero values when there is no causal effect. For this, point shuffling is employed as used in the frame of surrogate data. This correction is not specific to a particular measure and it is implemented here on measures based on state space reconstruction and information measures. The performance of the causality measures and their modifications is evaluated on simulated uncoupled and coupled dynamical systems and for different settings of embedding dimension, time series length and noise level. The corrected measures, and particularly the suggested corrected transfer entropy, turn out to stabilize at the zero level in the absence of causal effect and detect correctly the direction of information flow when it is present. The measures are also evaluated on electroencephalograms (EEG) for the detection of the information flow in the brain of an epileptic patient. The performance of the measures on EEG is interpreted, in view of the results from the simulation study.

An extension of transfer entropy, called partial transfer entropy, is proposed here to detect causal effects among observed interacting systems, and particularly distinguish among direct and indirect causal effects. The measure is presented for time series from three interacting systems, but can be extended for more time series. Partial transfer entropy is evaluated on simulated and real data and is compared with partial directed coherence, a linear causality measure defined in the frequency domain. The simulation study showed that both measures can identify linear and nonlinear causal effects for the correct parameters. Both measures are then tested on electroencephalograms of epileptic patients for the detection of the direction of the interaction among brain areas during the preictal, ictal and postictal states.

In a recent work we proposed the corrected transfer entropy (CTE), which reduces the bias in the estimation of transfer entropy (TE), a measure of Granger causality for bivariate time series making use of the conditional mutual information. An extension of TE to account for the presence of other time series is the partial TE (PTE). Here, we propose the correction of PTE, termed Corrected PTE (CPTE), in a similar way to CTE: time shifted surrogates are used in order to quantify and correct the bias, and the estimation of the involved entropies of high-dimensional variables is made with the method of k-nearest neighbors. CPTE is evaluated on coupled stochastic systems with both linear and nonlinear interactions. Finally, we apply CPTE to economic data and investigate whether we can detect the direct causal effects among economic variables.

Abstract. Information causality measures, ie transfer entropy and symbolic transfer entropy, are modified using the concept of surrogate data in order to identify correctly the presence and direction of causal effects. The measures are evaluated on multiple bivariate time ...

Complexity of industrial plants and their stringent environmental and safety regulations have necessitated early detection and isolation of process faults. All the existing fault isolation methods can be categorized into two general groups: model-based and data-based. Transfer entropy is a data-based method for measuring propagation direction of disturbance and finding its root cause. In this paper, a new transfer entropy-based method is proposed to isolate different process faults. The novelty of this paper lies in using the transfer entropy idea to generate distinct patterns of information flow among process variables, recognize their correlations in the context of the transferred information in any abnormal condition, and finally isolate different process faults. The experimental results clearly demonstrate the superiority of the proposed method to the conventional methods.

Transfer entropy (TE) captures the directed relationships between two variables. Partial transfer entropy (PTE) accounts for the presence of all confounding variables of a multivariate system and infers only about direct causality. However, the computation of partial transfer entropy involves high dimensional distributions and thus may not be robust in case of many variables. In this work, different variants of the partial transfer entropy are introduced, by building a reduced number of confounding variables based on different scenarios in terms of their interrelationships with the driving or response variable. Connectivity-based PTE variants utilizing the random forests (RF) methodology are evaluated on synthetic time series. The empirical findings indicate the superiority of the suggested variants over transfer entropy and partial transfer entropy, especially in the case of high dimensional systems. The above findings are further highlighted when applying the causality measures on...

Self-entropy (SE) and transfer entropy (TE) are widely utilized in biomedical signal processing to assess the information stored into a system and transferred from a source to a destination respectively. The study proposes a more specific definition of the SE, namely the conditional SE (CSE), and a more flexible definition of the TE based on joint TE (JTE), namely the conditional JTE (CJTE), for the analysis of information dynamics in multivariate time series. In a protocol evoking a gradual sympathetic activation and vagal withdrawal proportional to the magnitude of the orthostatic stimulus, such as the graded head-up tilt, we extracted the beat-to-beat spontaneous variability of heart period (HP), systolic arterial pressure (SAP) and respiratory activity (R) in 19 healthy subjects and we computed SE of HP, CSE of HP given SAP and R, JTE from SAP and R to HP, CJTE from SAP and R to HP given SAP and CJTE from SAP and R to HP given R. CSE of HP given SAP and R was significantly smaller than SE of HP and increased progressively with the amplitude of the stimulus, thus suggesting that dynamics internal to HP and unrelated to SAP and R, possibly linked to sympathetic activation evoked by head-up tilt, might play a role during the orthostatic challenge. While JTE from SAP and R to HP was independent of tilt table angle, CJTE from SAP and R to HP given R and from SAP and R to HP given SAP showed opposite trends with tilt table inclination, thus suggesting that the importance of the cardiac baroreflex increases and the relevance of the cardiopulmonary pathway decreases during head-up tilt. The study demonstrates the high specificity of CSE and the high flexibility of CJTE over real data and proves that they are particularly helpful in disentangling physiological mechanisms and in assessing their different contributions to the overall cardiovascular regulation.

Cross-frequency interactions, a form of oscillatory neural activity, are thought to play an essential role in the integration of distributed information in the brain. Indeed, phase-amplitude interactions are believed to allow for the transfer of information from large-scale brain networks, oscillating at low frequencies, to local, rapidly oscillating neural assemblies. A promising approach to estimating such interactions is the use of transfer entropy (TE), a non-linear, information-theory-based effective connectivity measure. The conventional method involves feeding instantaneous phase and amplitude time series, extracted at the target frequencies, to a TE estimator. In this work, we propose that the problem of directed phase-amplitude interaction detection is recast as a phase TE estimation problem, under the hypothesis that estimating TE from data of the same nature, i.e., two phase time series, will improve the robustness to the common confounding factors that affect connectivity measures, such as the presence of high noise levels. We implement our proposal using a kernel-based TE estimator, defined in terms of Renyi's α entropy, which has successfully been used to compute single-trial phase TE. We tested our approach on the synthetic data generated through a simulation model capable of producing a time series with directed phase-amplitude interactions at two given frequencies, and on EEG data from a cognitive task designed to activate working memory, a memory system whose underpinning mechanisms are thought to include phase-amplitude couplings. Our proposal detected statistically significant interactions between the simulated signals at the desired frequencies for the synthetic data, identifying the correct direction of the interaction. It also displayed higher robustness to noise than the alternative methods. The results attained for the working memory data showed that the proposed approach codes connectivity patterns based on directed phaseamplitude interactions, that allow for the different cognitive load levels of the working memory task to be differentiated.

Neural oscillations are present in the brain at different spatial and temporal scales, and they are linked to several cognitive functions. Furthermore, the information carried by their phases is fundamental for the coordination of anatomically distributed processing in the brain. The concept of phase transfer entropy refers to an information theory-based measure of directed connectivity among neural oscillations that allows studying such distributed processes. Phase TE is commonly obtained from probability estimations carried out over data from multiple trials, which bars its use as a characterization strategy in brain-computer interfaces. In this work, we propose a novel methodology to estimate TE between single pairs of instantaneous phase time series. Our approach combines a kernel-based TE estimator defined in terms of Renyi's α entropy, which sidesteps the need for probability distribution computation with phase time series obtained by complex filtering the neural signals. Besides, a kernel-alignment-based relevance analysis is added to highlight relevant features from effective connectivity-based representation supporting further classification stages in EEG-based brain-computer interface systems. Our proposal is tested on simulated coupled data and two publicly available databases containing EEG signals recorded under motor imagery and visual working memory paradigms. Attained results demonstrate how the introduced effective connectivity succeeds in detecting the interactions present in the data for the former, with statistically significant results around the frequencies of interest. It also reflects differences in coupling strength, is robust to realistic noise and signal mixing levels, and captures bidirectional interactions of localized frequency content. Obtained results for the motor imagery and working memory databases show that our approach, combined with the relevance analysis strategy, codes discriminant spatial and frequencydependent patterns for the different conditions in each experimental paradigm, with classification performances that do well in comparison with those of alternative methods of similar nature.

Neural oscillations are present in the brain at different spatial and temporal scales, and they are linked to several cognitive functions. Furthermore, the information carried by their phases is fundamental for the coordination of anatomically distributed processing in the brain. The concept of phase transfer entropy refers to an information theory-based measure of directed connectivity among neural oscillations that allows studying such distributed processes. Phase TE is commonly obtained from probability estimations carried out over data from multiple trials, which bars its use as a characterization strategy in brain–computer interfaces. In this work, we propose a novel methodology to estimate TE between single pairs of instantaneous phase time series. Our approach combines a kernel-based TE estimator defined in terms of Renyi’s α entropy, which sidesteps the need for probability distribution computation with phase time series obtained by complex filtering the neural signals. Besi...

Cross-frequency interactions, a form of oscillatory neural activity, are thought to play an essential role in the integration of distributed information in the brain. Indeed, phase-amplitude interactions are believed to allow for the transfer of information from large-scale brain networks, oscillating at low frequencies, to local, rapidly oscillating neural assemblies. A promising approach to estimating such interactions is the use of transfer entropy (TE), a non-linear, information-theory-based effective connectivity measure. The conventional method involves feeding instantaneous phase and amplitude time series, extracted at the target frequencies, to a TE estimator. In this work, we propose that the problem of directed phase-amplitude interaction detection is recast as a phase TE estimation problem, under the hypothesis that estimating TE from data of the same nature, i.e., two phase time series, will improve the robustness to the common confounding factors that affect connectivit...

Nature is full of random networks of complex topology describing such apparently disparate systems as biological, economical or informatical ones. Their most characteristic feature is the apparent scale-free character of interconnections between nodes. Using an information theory approach, we show that maximalization of information entropy leads to a wide spectrum of possible types of distributions including, in the case of nonextensive information entropy, the power-like scale-free distributions characteristic of complex systems.

We introduce a novel measure, Fisher transfer entropy (FTE), which quantifies a gain in sensitivity to a control parameter of a state transition, in the context of another observable source. The new measure captures both transient and contextual qualities of transfer entropy and the sensitivity characteristics of Fisher information. FTE is exemplified for a ferromagnetic two-dimensional lattice Ising model with Glauber dynamics and is shown to diverge at the critical point.

Entropy measures in their various incarnations play an important role in the study of stochastic time series providing important insights into both the correlative and the causative structure of the stochastic relationships between the individual components of a system. Recent applications of entropic techniques and their linear progenitors such as Pearson correlations and Granger causality have have included both normal as well as critical periods in a system's dynamical evolution. Here I measure the entropy, Pearson correlation and transfer entropy of the intra-day price changes of the Dow Jones Industrial Average (DJIA) in the period immediately leading up to and including the Asian financial crisis and subsequent mini-crash of the DJIA on the 27 th October 1997. I use a novel variation of transfer entropy that dynamically adjusts to the arrival rate of individual prices and does not require the binning of data to show that quite different relationships emerge from those given by the conventional Pearson correlations between equities. These preliminary results illustrate how this modified form of the TE compares to results using Pearson correlation.

There is growing evidence that for a range of dynamical systems featuring complex interactions between large ensembles of interacting elements, mutual information peaks at order/disorder phase transitions. We conjecture that, by contrast, information flow in such systems will generally peak strictly on the disordered side of a phase transition. This conjecture is verified for a ferromagnetic 2d lattice Ising model with Glauber dynamics and a transfer entropy-based measure of system-wide information flow. Implications of the conjecture are considered; in particular, that for a complex dynamical system in the process of transitioning from disordered to ordered dynamics (a mechanism implicated, for example, in financial market crashes and the onset of some types of epileptic seizures), information dynamics may be able to predict an imminent transition.

As a causality criterion we propose the conditional relative entropy. The relationship with information theoretic functionals mutual information and entropy is established. The conditional relative entropy criterion is compared with 3 well-established techniques for causality detection: 'Sims', 'Geweke-Meese-Dent' and 'Granger'. It is shown that the conditional relative entropy, as opposed to these 3 criteria, is sensitive to0. non-linear causal relationships. All results are illustrated on real-world time series of human gait.

The concept of "Industry 5.0" is driving significant changes in the production of chemical products and energy, promoting a shift towards a decarbonized and circular economy. Digitalization, robotics, communications, and artificial intelligence (AI) play crucial roles in fostering the development of necessary technological innovations and enhancing intelligent process control. The application of machine deep learning (ML) yields robust, field-neutral solutions for regression prediction objectives, but it is limited in its capacity to address innovative questions that involve causation and counterfactual analysis. This paper presents a proposed application of Bayesian networks (BN) for structural causal modeling (SCM) in the context of manufacturing plants. A critical feature of SCM modeling is its capacity to integrate extensive prior structural knowledge derived from fundamental chemical engineering principles with structures inferred from experimental data obtained from manufacturing plants. The acquired SCM facilitates the forecasting of causal relationships, the simulation of intervention strategies, and the generation of counterfactual responses essential for process innovations and intelligent process management. The SCM model is presented as a tool for examining causality and control in the intricate Tennessee-Eastman process.

Data from social media provide unprecedented opportunities to investigate the processes that govern the dynamics of collective social phenomena. We consider an information theoretical approach to define and measure the temporal and structural signatures typical of collective social events as they arise and gain prominence. We use the symbolic transfer entropy analysis of microblogging time series to extract directed networks of influence among geolocalized subunits in social systems. This methodology captures the emergence of system-level dynamics close to the onset of socially relevant collective phenomena. The framework is validated against a detailed empirical analysis of five case studies. In particular, we identify a change in the characteristic time scale of the information transfer that flags the onset of information-driven collective phenomena. Furthermore, our approach identifies an order-disorder transition in the directed network of influence between social subunits. In t...

We propose a novel measure to detect temporal ordering in the activity of individual neurons in a local network, which is thought to be a hallmark of activity-dependent synaptic modifications during learning. The measure, called Causal Entropy, is based on the time-adaptive detection of asymmetries in the relative temporal patterning between neuronal pairs. We characterize properties of the measure on both simulated data and experimental multiunit recordings of hippocampal neurons from the awake, behaving rat, and show that the metric can more readily detect those asymmetries than standard cross correlation-based techniques, especially since the temporal sensitivity of causal entropy can detect such changes rapidly and dynamically.