Functional Isolation Forest

2019

This paper deals with the problem of modelling and monitoring the fault-free states of an industrial process without complete knowledge about the entire machine components. The aim thereby is to automatically detect the deviations in performance as fault symptoms. For that type of data-based modelling, the algorithms of clustering are selected with an emphasis on the computational load and application complexity. Kohonen neural networks (self-organizing maps) are found suitable for the task due to the ability to efficiently operate on high dimensional data and because of their robustness against uncertainties. They reveal drawbacks from the perspective of identifying the deviating variable in the input space. A novel structure is designed to solve this dilemma by combining multi one-dimensional domains and their statistical relationships, where Kohonen and Bayesian algorithms would be directly applicable. The structure is introduced and applied to simulate the human supervisors in t...

IEEE Access

Historically, anomalies detection was an important issue for industrial applications such as the detection of a manufacturing failure or defect. It is still a current topic that tries to meet the ever increasing demand in different fields such as intrusion detection, fraud detection, ecosystem change detection or event detection in sensor networks. That's why anomalies detection remains a research topic of great interest for various research communities. In this paper, we focused on Isolation Forest (IForest), a well known, efficient anomalies detection algorithm. We provided a deep and complete view on IForest. We evaluated the impact of its input parameters (number of trees, sample size and decision threshold) on the efficiency of the detection and on the execution time. We discussed the benefit of including some anomalies into the training phase. To address the limits of IForest, we performed different experiments on commonly used real datasets and also on synthetic datasets with non trivial distributions. We designed multidimensional datasets where anomalies are carried by several dimensions simultaneously. Moreover, we used a varying density and distance between anomalies and normal data, for a variable similarity between these two data classes. We compared the performance of IForest against its improved version called Extended IForest. Finally, we designed and validated a new extension of IForest, based on the different individual trees decisions instead of a global forest decision that we call Majority Voting IForest (MVIForest). The experiments show that MVIForest has a shorter execution time than IForest, with almost the same accuracy. INDEX TERMS anomaly detection, isolation-based, Isolation Forest, outlier, survey I. INTRODUCTION 1 A NOMALIES are often defined as elements with dif-2 ferent behavior compared to normal data. Anomalies 3 detection is an interesting issue widely studied by different 4 research communities: statistics, data mining, machine learn-5 ing and more recently deep learning. In fact, it has many 6 real-world applications such as intrusion detection, astron-7 omy, finance or cybersecurity. A fast and efficient anomalies 8 detection can avoid huge economic losses, natural disasters 9 and even save human lives. 10 Many anomalies detection techniques based on different 11 approaches are proposed in the literature. The most known re-12 views of anomalies detection existing techniques are [1], [4], 13 [5] and [19]. They identify the main following approaches: statistical approach, clustering and nearest neighbors. These 15 two latter categories are often either density based (such as 16 Local Outlier Factory (LOF) [3] and Density-Based Spatial

arXiv (Cornell University), 2024

Functional Isolation Forest (FIF) is a recent state-of-the-art Anomaly Detection (AD) algorithm designed for functional data. It relies on a tree partition procedure where an abnormality score is computed by projecting each curve observation on a drawn dictionary through a linear inner product. Such linear inner product and the dictionary are a priori choices that highly influence the algorithm's performances and might lead to unreliable results, particularly with complex datasets. This work addresses these challenges by introducing Signature Isolation Forest, a novel AD algorithm class leveraging the rough path theory's signature transform. Our objective is to remove the constraints imposed by FIF through the proposition of two algorithms which specifically target the linearity of the FIF inner product and the choice of the dictionary. We provide several numerical experiments, including a real-world applications benchmark showing the relevance of our methods.

Data-Centric Engineering

Population-based structural health monitoring (PBSHM) provides a means of accounting for inter-turbine correlations when solving the problem of wind farm anomaly detection. Across a wind farm, where a group of structures (turbines) is placed in close vicinity to each other, the environmental conditions and, thus, structural behavior vary in a spatiotemporal manner. Spatiotemporal trends are often overlooked in the existing data-based wind farm anomaly detection methods, because most current methods are designed for individual structures, that is, detecting anomalous behavior of a turbine based on the past behavior of the same turbine. In contrast, the idea of PBSHM involves sharing data across a population of structures and capturing the interactions between structures. This paper proposes a population-based anomaly detection method, specifically for a localized population of structures, which accounts for the spatiotemporal correlations in structural behavior. A case study from an ...

2005

We address the problem of online detection of unanticipated modes of mechanical failure given a small set of time series under normal conditions, with the requirement that the anomaly detection model be manually verifiable and modifiable. We specify a set of time series features, which are linear combinations of the current and past values, and model the allowed feature values by a sequence of minimal bounding boxes containing all of the training trajectories. The model can be constructed in O(n log n) time. If there are at most three features, the model can be displayed graphically for verification, otherwise a table is used. Test time is O(n) with a guaranteed upper bound on computation time for each test point. The model compares favorably with anomaly detection algorithms based on Euclidean distance and dynamic time warping on the Space Shuttle Marrotta fuel control valve data set.

Advanced Engineering Informatics, 2020

Performance analysis of the existing mechanical products is critical to identifying design defects and improving product reliability. With the advances of information technologies, product operating data collected through continuous condition monitoring (CM) serve as main sources for analysis of performance and detection of anomaly. Most of the existing anomaly detection methods, however, are not effective when CM data are very high dimensional, leading to poor quality of assessment results. Besides, the effects of multiple operating conditions on anomaly detection are seldom considered in these existing methods. To solve these problems, an integrated approach for anomaly detection and critical behavioral attributes identification based on CM data is developed in this research. Gaussian mixed model GMM) is employed to develop a method for clustering of operating conditions. Isolation forest (iForest) method is used to detect anomaly instances, and further to identify the critical attributes related to product performance degradation. The effectiveness of the developed approach is demonstrated by an application with collected operating data of a wind turbine.

IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020

Timely detection of abrupt anomalies is crucial for real-time monitoring and security of modern systems producing high-dimensional data. With this goal, we propose effective and scalable algorithms. Proposed algorithms are nonparametric as both the nominal and anomalous multivariate data distributions are assumed unknown. We extract useful univariate summary statistics and perform anomaly detection in a single-dimensional space. We model anomalies as persistent outliers and propose to detect them via a cumulative sum-like algorithm. In case the observed data have a low intrinsic dimensionality, we learn a submanifold in which the nominal data are embedded and evaluate whether the sequentially acquired data persistently deviate from the nominal submanifold. Further, in the general case, we learn an acceptance region for nominal data via Geometric Entropy Minimization and evaluate whether the sequentially observed data persistently fall outside the acceptance region. We provide an asymptotic lower bound and an asymptotic approximation for the average false alarm period of the proposed algorithm. Moreover, we provide a sufficient condition to asymptotically guarantee that the decision statistic of the proposed algorithm does not diverge in the absence of anomalies. Experiments illustrate the effectiveness of the proposed schemes in quick and accurate anomaly detection in high-dimensional settings.

IRJET, 2020

Anomaly detection (also outlier detection [1]) is the identification of rare items, events or observations which raise suspicions by differing significantly from the majority of the data.[1] Typically the anomalous items will translate to some kind of problem such as bank fraud, a structural defect, medical problems or errors in a text. Anomalies are also referred to as outliers, novelties, noise, deviations and exceptions. [2] In particular, in the context of abuse and network intrusion detection, the interesting objects are often not rare objects, but unexpected bursts in activity. This pattern does not adhere to the common statistical definition of an outlier as a rare object, and many outlier detection methods (in particular unsupervised methods) will fail on such data, unless it has been aggregated appropriately. Instead, a cluster analysis algorithm may be able to detect the micro clusters formed by these patterns. [3] Three broad categories of anomaly detection techniques exist.[4] Unsupervised anomaly detection techniques detect anomalies in an unlabeled test data set under the assumption that the majority of the instances in the data set are normal by looking for instances that seem to fit least to the remainder of the data set. Supervised anomaly detection techniques require a data set that has been labeled as "normal" and "abnormal" and involves training a classifier (the key difference to many other statistical classification problems is the inherent unbalanced nature of outlier detection). Semi-supervised anomaly detection techniques construct a model representing normal behavior from a given normal training data set, and then test the likelihood of a test instance to be generated by the learnt model.

2008 IEEE International Conference on Data Mining Workshops, 2008

Anomaly detection in multivariate time series is an important data mining task with applications to ecosystem modeling, network traffic monitoring, medical diagnosis, and other domains. This paper presents a robust algorithm for detecting anomalies in noisy multivariate time series data by employing a kernel matrix alignment method to capture the dependence relationships among variables in the time series. Anomalies are found by performing a random walk traversal on the graph induced by the aligned kernel matrix. We show that the algorithm is flexible enough to handle different types of time series anomalies including subsequence-based and local anomalies. Our framework can also be used to characterize the anomalies found in a target time series in terms of the anomalies present in other time series. We have performed extensive experiments to empirically demonstrate the effectiveness of our algorithm. A case study is also presented to illustrate the ability of the algorithm to detect ecosystem disturbances in Earth science data.

Anomaly detection in multivariate time series is an important data mining task with applications to ecosystem modeling, network traffic monitoring, medical diagnosis, and other domains. This paper presents a robust algorithm for detecting anomalies in noisy mul-tivariate time series data by employing a kernel matrix alignment method to capture the dependence relationships among variables in the time series. Anomalies are found by performing a random walk traversal on the graph induced by the aligned kernel matrix. We show that the algorithm is flexible enough to handle different types of time series anomalies including subsequence-based and local anomalies. Our framework can also be used to characterize the anomalies found in a target time series in terms of the anomalies present in other time series. We have performed extensive experiments to empirically demonstrate the effectiveness of our algorithm. A case study is also presented to illustrate the ability of the algorithm to detect ecosystem disturbances in Earth science data.