Distance-Based Outlier Detection: Consolidation and Renewed Bearing

Journal of Computer Science, 2007

A fast distance-based algorithm for outlier detection will be proposed. It was found that the proposed algorithm reduced the number of distance calculations compared to the nestedloop algorithm. Test results were performed on different well-known data sets. The results showed that the proposed algorithm gave a reasonable amount of CPU time saving.

2000

Outlier detection is a process to identify rare or abnormal observations in the data. Distance based techniques can find only global outliers whereas density based techniques can capture local outliers based on their neighborhood densities. We present a method to find outliers using 'commute time distance' which is computed from a random walk of a weighted graph derived from the data similarity matrix. Unlike Euclidian distance, commute time distance between two nodes captures both the distance between them and their neighborhood densities. We show by analysis and experiments that using this measure, we can capture both global and local outliers with just a distance based method. Moreover, to avoid the O(n 3) direct computation of commute time distance, a subspace approximation combined with pruning technique in a distance based outlier detection reduce the time significantly while preserving the outlier ranking.

IEEE Access

Detecting outliers is a significant problem that has been studied in various research and application areas. Researchers continue to design robust schemes to provide solutions to detect outliers efficiently. In this survey, we present a comprehensive and organized review of the progress of outlier detection methods from 2000-2019. Firstly, we offer the fundamental concepts of outlier detection and then categorize them into different techniques from diverse outlier detection techniques such as distance-, clustering-, density-, ensemble-, and learning-based methods. In each category, we introduce some state-ofthe-art outlier detection methods and further discuss them in detail in terms of their performance. Secondly, we delineate their pros, cons, and challenges to provide researchers with a concise overview of each technique and recommend solutions and possible research directions. This paper gives current progress of outlier detection techniques and provides a better understanding of the different outlier detection methods. The open research issues and challenges at the end will provide researchers with a clear path for the future of outlier detection methods.

Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining - KDD '11, 2011

The problem of distance-based outlier detection is difficult to solve efficiently in very large datasets because of potential quadratic time complexity. We address this problem and develop sequential and distributed algorithms that are significantly more efficient than state-of-the-art methods while still guaranteeing the same outliers. By combining simple but effective indexing and disk block accessing techniques, we have developed a sequential algorithm iOrca that is up to an order-of-magnitude faster than the state-of-the-art. The indexing scheme is based on sorting the data points in order of increasing distance from a fixed reference point and then accessing those points based on this sorted order. To speed up the basic outlier detection technique, we develop two distributed algorithms (DOoR and iDOoR) for modern distributed multi-core clusters of machines, connected on a ring topology. The first algorithm passes data blocks from each machine around the ring, incrementally updating the nearest neighbors of the points passed. By maintaining a cutoff threshold, it is able to prune a large number of points in a distributed fashion. The second distributed algorithm extends this basic idea with the indexing scheme discussed earlier. In our experiments, both distributed algorithms exhibit significant improvements compared to the state-of-the-art distributed method .

Proceedings of the Thirteenth ACM conference on Information and knowledge management - CIKM '04, 2004

One person's noise is another person's signal". Outlier detection is used to clean up datasets and also to discover useful anomalies, such as criminal activities in electronic commerce, computer intrusion attacks, terrorist threats, agricultural pest infestations, etc. Thus, outlier detection is critically important in the information-based society. This paper focuses on finding outliers in large datasets using distance-based methods. First, to speedup outlier detections, we revise Knorr and Ng's distance-based outlier definition; second, a vertical data structure, instead of traditional horizontal structures, is adopted to facilitate efficient outlier detection further. We tested our methods against national hockey league dataset and show an order of magnitude of speed improvement compared to the contemporary distance-based outlier detection approaches.

Indonesian Journal of Electrical Engineering and Computer Science, 2021

Outliers are eccentric data points with anomalous nature. Clustering with outliers has received a lot of attention in the data processing community. But, they inordinately affect the quality of the results obtained in case of popular clustering algorithms during the process of finding an optimal solution. In this work, we propose a novel method to classify the data points with grouping characteristics as either an outlier or not. We use both distance and density of a particular data point with respect to the rest of the data points for this process. Distances are used to find the points at the extremities while the densities are used to identify the data points at the sparsest spaces. Further, every data model has to take into account the aspect of generalization in order to work robustly even in out of the box situations. Hence, our approach provides a generalization aspect to the model. The accuracy of the proposed work is measured using area under curve (AUC) was found the highes...

2017 IEEE 33rd International Conference on Data Engineering (ICDE), 2017

As datasets increase radically in size, highly scalable algorithms leveraging modern distributed infrastructures need to be developed for detecting outliers in massive datasets. In this work, we present the first distributed distance-based outlier detection approach using the MapReduce-based infrastructure, called DOD. DOD features a single-pass execution framework that minimizes communication overhead. Furthermore, DOD overturns two fundamental assumptions widely adopted in the distributed analytics literature, namely cardinality-based load balancing and one algorithm for all data. The multi-tactic strategy of DOD achieves a truly balanced workload by taking into account the data characteristics in data partitioning and assigns most appropriate algorithm for each partition based on our theoretical cost models established for distinct classes of detection algorithms. Thus, DOD effectively minimizes the end-to-end execution time. Our experimental study confirms the efficiency of DOD and its scalability to terabytes of data, beating the baseline solutions by a factor of 20x.

Outlier identification previously, high-dimensional information displays Different tests coming about because of the " curse of dimensionality ". An prevailing perspective may be that separation concentration, i.e., the propensity from claiming distances over high-dimensional information on turned into indiscernible, Hinders those identification about outliers by making distance-based strategies name at focuses as very nearly just as beneficial outliers. In this paper, we gatherings give proof supporting the assumption that such a see will be a really simple, toward demonstrating that distance-based strategies could. Handle All the more differentiating outlier scores previously, high-dimensional settings. Furthermore, we demonstrate that helter skelter dimensionality could have. An alternate impact, toward reexamining those ideas about opposite closest neighbours in the unsupervised outlier-detection connection. Namely, it might have been as of late watched that those dissemination about points' reverse-neighbour tallies turns into skewed on secondary dimensions, bringing about. The wonder known as hubness. We furnish knowledge under how a few focuses (antihubs) show up extremely rarely to k-NN records about. Different points, What's more clarify those association between antihubs, outliers, and existing unsupervised outlier-detection systems .Toward. Assessing that excellent k-NN method that angle-based method outlined to high-dimensional data, those density-based nearby. Outlier component What's more impacted outlierness methods, also antihubs-based strategies on Different manufactured Also real-world information sets, we offer novel knowledge under the convenience about opposite neighbour tallies previously, unsupervised outlier identification.

2014

Uncertain data management, querying and mining have become important because the majority of real world data is accompanied with uncertainty these days. Uncertainty in data is often caused by the deficiency in underlying data collecting equipments or sometimes manually introduced to preserve data privacy. The uncertainty information in the data is useful and can be used to improve the quality of the underlying results. Therefore in this dissertation, three problems are being solved related to outlier detection on uncertain data. 1) Distancebased outlier detection on uncertain data: In this research, we give a novel definition of distance-based outliers on uncertain data. Since the distance probability computation is expensive, a cell-based approach is proposed to index the dataset objects and to speed up the outlier detection process. The cell-based approach identifies and prunes the cells containing only inliers based on its bounds on outlier score (#D-neighbors). Similarly it can ...

edventure.ntu.edu.sg