A Fast Distance-Based Algorithm to Detect Outliers

Indonesian Journal of Electrical Engineering and Computer Science, 2021

Outliers are eccentric data points with anomalous nature. Clustering with outliers has received a lot of attention in the data processing community. But, they inordinately affect the quality of the results obtained in case of popular clustering algorithms during the process of finding an optimal solution. In this work, we propose a novel method to classify the data points with grouping characteristics as either an outlier or not. We use both distance and density of a particular data point with respect to the rest of the data points for this process. Distances are used to find the points at the extremities while the densities are used to identify the data points at the sparsest spaces. Further, every data model has to take into account the aspect of generalization in order to work robustly even in out of the box situations. Hence, our approach provides a generalization aspect to the model. The accuracy of the proposed work is measured using area under curve (AUC) was found the highes...

International Journal of Computer Applications, 2013

Outlier detection is an extremely important problem with direct application in a wide variety of domains. A key challenge with outlier detection is that it is not a wellformulated problem like clustering. In this paper, discussion on different techniques and then comparison by analyzing their different aspects, essentially, time complexity. Every unique problem formulation entails a different approach, resulting in a huge literature on outlier detection techniques. Several techniques have been proposed to target a particular application domain. The classification of outlier detection techniques based on the applied knowledge discipline provides an idea of the research done by different communities and also highlights the unexplored research avenues for the outlier detection problem. Discussed of the behavior of different techniques will be done, in this paper, with respect to the nature. The feasibility of a technique in a particular problem setting also depends on other constraints. For example, Statistical techniques assume knowledge about the underlying distribution characteristics of the data. Distance based techniques are typically expensive and hence are not applied in scenarios where computational complexity is an important issue.

— as there is an increasing demand of data, outlier detection is coming across as a popular field of research. Outlier is stated as an observation which is dissimilar from the other observations present in the data set. It is advantageous in the fields like medical industry, crime detection, fraudulent transaction, public safety etc. Outlier can be learnt in different fields like big data, time series data, high dimension data, biological data, uncertain data and many more. Most of the time 10% of the whole sample data set is incorrect, not accessible or missing sometimes. This paper studies and compares the popular outlier detection algorithms namely, Cluster based outlier detection, Distance based outlier detection and Density based outlier detection. Comparative study of these outlier detection techniques is performed to find out most efficient outlier detection method for calculation of the outlier.

Proceedings of the Thirteenth ACM conference on Information and knowledge management - CIKM '04, 2004

One person's noise is another person's signal". Outlier detection is used to clean up datasets and also to discover useful anomalies, such as criminal activities in electronic commerce, computer intrusion attacks, terrorist threats, agricultural pest infestations, etc. Thus, outlier detection is critically important in the information-based society. This paper focuses on finding outliers in large datasets using distance-based methods. First, to speedup outlier detections, we revise Knorr and Ng's distance-based outlier definition; second, a vertical data structure, instead of traditional horizontal structures, is adopted to facilitate efficient outlier detection further. We tested our methods against national hockey league dataset and show an order of magnitude of speed improvement compared to the contemporary distance-based outlier detection approaches.

This paper presents a new approach for detecting outliers by introducing the notion of object’s proximity. The main idea is that normal point has similar characteristics with several neighbors. So the point in not an outlier if it has a high degree of proximity and its neighbors are several. The performance of this approach is illustrated through real datasets.

Lecture Notes in Computer Science, 2011

In this paper, we propose a nearest neighbor based outlier detection algorithm, NDoT . We introduce a parameter termed as N earest N eighbor F actor (NNF ) to measure the degree of outlierness of a point with respect to its neighborhood. Unlike the previous outlier detection methods NDoT works by a voting mechanism. Voting mechanism binarizes the decision compared to the top-N style of algorithms. We evaluate our method experimentally and compare results of NDoT with a classical outlier detection method LOF and a recently proposed method LDOF . Experimental results demonstrate that NDoT outperforms LDOF and is comparable with LOF .

Proceedings of The Vldb Endowment, 2010

Detecting outliers in data is an important problem with interesting applications in a myriad of domains ranging from data cleaning to financial fraud detection and from network intrusion detection to clinical diagnosis of diseases. Over the last decade of research, distance-based outlier detection algorithms have emerged as a viable, scalable, parameter-free alternative to the more traditional statistical approaches.

International Journal of Computer Applications, 2012

Outlier detection is a fundamental issue in data mining, specifically it has been used to detect and remove anomalous objects from data.mining. The proposed approach to detect outlier includes three methods which are clustering, pruning and computing outlier score. For clustering k-means algorithm is used which partition the dataset into given number of clusters. In pruning, based on some distance measure, points which are closed to centroid of each cluster are pruned. For the unpruned points, local distance based outlier factor (LDOF) measure is calculated. A measure called LDOF, tells how much a point is deviating from its neighbors. The high LDOF value of a point indicates that the point is deviating more from its neighbors and probably it may be an outlier.

Outlier Detection is one of the most important topics in data mining and knowledge discovery in databases (KDD). It is to find a methodology to detect instances in a dataset that do not conform to an established normal behavior. There are some researches to find outliers, for example Local Outlier Factor

Outlier (or anomaly) detection is an important problem for many domains, including fraud detection, risk analysis, network intrusion and medical diagnosis, and the discovery of significant outliers is becoming an integral aspect of data mining. This paper presents CURIO, a novel algorithm that uses quantisation and implied distance metrics to provide a fast algorithm that is linear for the number of objects and only requires two sequential scans of disk resident datasets. CURIO includes a novel direct quantisation technique and the explicit discovery of outlier clusters. Moreover, a major attribute of CURIO is its simplicity and economy with respect to algorithm, memory footprint and data structures.