Academia.eduAcademia.edu

Outline

Title
Abstract
Introduction
Naive Index-Based Method
Two-Way Search Method
Micro-Cluster-Based Method
Related Work
Conclusion
References
All Topics
Computer Science
Machine Learning

Ranking Outliers Using Symmetric Neighborhood Relationship

Profile image of Jiawei HanJiawei Han

2006, Advances in Knowledge Discovery and Data Mining

https://doi.org/10.1007/11731139_68
visibility

…

description

10 pages

link

1 file

Abstract

Mining outliers in database is to find exceptional objects that deviate from the rest of the data set. Besides classical outlier analysis algorithms, recent studies have focused on mining local outliers, i.e., the outliers that have density distribution significantly different from their neighborhood. The estimation of density distribution at the location of an object has so far been based on the density distribution of its k-nearest neighbors [2, 11]. However, when outliers are in the location where the density distributions in the neighborhood are significantly different, for example, in the case of objects from a sparse cluster close to a denser cluster, this may result in wrong estimation. To avoid this problem, here we propose a simple but effective measure on local outliers based on a symmetric neighborhood relationship. The proposed measure considers both neighbors and reverse neighbors of an object when estimating its density distribution. As a result, outliers so discovered are more meaningful. To compute such local outliers efficiently, several mining algorithms are developed that detects top-n outliers based on our definition. A comprehensive performance evaluation and analysis shows that our methods are not only efficient in the computation but also more effective in ranking outliers.

Related papers

Outlier Detection Score Based on Ordered Distance Difference
Krung Sinapiromsaran

Outlier Detection is one of the most important topics in data mining and knowledge discovery in databases (KDD). It is to find a methodology to detect instances in a dataset that do not conform to an established normal behavior. There are some researches to find outliers, for example Local Outlier Factor

View PDFchevron_right
A simple, effective distance and density based outlier detection algorithm
Udai Agarwal

Indonesian Journal of Electrical Engineering and Computer Science, 2021

Outliers are eccentric data points with anomalous nature. Clustering with outliers has received a lot of attention in the data processing community. But, they inordinately affect the quality of the results obtained in case of popular clustering algorithms during the process of finding an optimal solution. In this work, we propose a novel method to classify the data points with grouping characteristics as either an outlier or not. We use both distance and density of a particular data point with respect to the rest of the data points for this process. Distances are used to find the points at the extremities while the densities are used to identify the data points at the sparsest spaces. Further, every data model has to take into account the aspect of generalization in order to work robustly even in out of the box situations. Hence, our approach provides a generalization aspect to the model. The accuracy of the proposed work is measured using area under curve (AUC) was found the highes...

View PDFchevron_right
Outlier Detection based on Distance: Reverse Nearest Neighbors
IJIRST - International Journal for Innovative Research in Science and Technology

Outlier identification previously, high-dimensional information displays Different tests coming about because of the " curse of dimensionality ". An prevailing perspective may be that separation concentration, i.e., the propensity from claiming distances over high-dimensional information on turned into indiscernible, Hinders those identification about outliers by making distance-based strategies name at focuses as very nearly just as beneficial outliers. In this paper, we gatherings give proof supporting the assumption that such a see will be a really simple, toward demonstrating that distance-based strategies could. Handle All the more differentiating outlier scores previously, high-dimensional settings. Furthermore, we demonstrate that helter skelter dimensionality could have. An alternate impact, toward reexamining those ideas about opposite closest neighbours in the unsupervised outlier-detection connection. Namely, it might have been as of late watched that those dissemination about points' reverse-neighbour tallies turns into skewed on secondary dimensions, bringing about. The wonder known as hubness. We furnish knowledge under how a few focuses (antihubs) show up extremely rarely to k-NN records about. Different points, What's more clarify those association between antihubs, outliers, and existing unsupervised outlier-detection systems .Toward. Assessing that excellent k-NN method that angle-based method outlined to high-dimensional data, those density-based nearby. Outlier component What's more impacted outlierness methods, also antihubs-based strategies on Different manufactured Also real-world information sets, we offer novel knowledge under the convenience about opposite neighbour tallies previously, unsupervised outlier identification.

View PDFchevron_right
Class outliers mining: Distance-based approach
Motaz Saad

International Journal of Intelligent Systems and …

View PDFchevron_right
A comparative Study of Outlier Mining and Class Outlier Mining
Motaz Saad

Computer Science Letters, 2009

View PDFchevron_right
A Survey for Different Approaches of Outlier Detection in Data Mining
krishna patel

— Outlier is defined as an event that deviates too much from other events. The identification of outlier can lead to the discovery of useful and meaningful knowledge. Outlier means it's happen at some time it's not regular activity. Research about Detection of Outlier has been extensively studies in the past decade. However, most existing research focused on the algorithm based on specific knowledge, compared with outlier detection approach is still rare. In this paper mainly focused on different kind of outlier detection approaches and compares it's prone and cones. In this paper we mainly distribute of outlier detection approach in two parts classic outlier approach and spatial outlier approach. The classical outlier approach identifies outlier in real transaction dataset, which can be grouped into statistical approach, distance approach, deviation approach, and density approach. The spatial outlier approach detect outlier based on spatial dataset are different from transaction data, which can be categorized into spaced approach and graph approach. Finally, the comparison of outlier detection approaches.

View PDFchevron_right
Similarity- based approach for outlier detection
Amina Dik

This paper presents a new approach for detecting outliers by introducing the notion of object’s proximity. The main idea is that normal point has similar characteristics with several neighbors. So the point in not an outlier if it has a high degree of proximity and its neighbors are several. The performance of this approach is illustrated through real datasets.

View PDFchevron_right
Density-Based Approach for Outlier Detection and Removal
Sakshi Saxena

Lecture Notes in Electrical Engineering, 2018

This paper represents an algorithm for performing clustering and outlier detection simultaneously. As research says, clustering and outlier (anomaly) detection are not separate problems but they are co-related. So our algorithm provides a generalized solution for outlier detection as per application. It takes some threshold values as input, applies K-means algorithm for initial clustering and based on threshold values, outliers are detected. This approach is not strict to number of clusters k, but applies re-clustering where required. It helps to find local as well as global outliers of dataset. The results can be customized by varying the values of threshold limits. The algorithm works in two phases, first phase provides initial clustering using K-Means and second phase helps to find outliers.

View PDFchevron_right
CURIO : A Fast Outlier and Outlier Cluster Detection Algorithm for Large Datasets
David Powers

Outlier (or anomaly) detection is an important problem for many domains, including fraud detection, risk analysis, network intrusion and medical diagnosis, and the discovery of significant outliers is becoming an integral aspect of data mining. This paper presents CURIO, a novel algorithm that uses quantisation and implied distance metrics to provide a fast algorithm that is linear for the number of objects and only requires two sequential scans of disk resident datasets. CURIO includes a novel direct quantisation technique and the explicit discovery of outlier clusters. Moreover, a major attribute of CURIO is its simplicity and economy with respect to algorithm, memory footprint and data structures.

View PDFchevron_right
A Comparative Study of Cluster Based Outlier Detection, Distance Based Outlier Detection and Density Based Outlier Detection Techniques
Harshada C Mandhare

— as there is an increasing demand of data, outlier detection is coming across as a popular field of research. Outlier is stated as an observation which is dissimilar from the other observations present in the data set. It is advantageous in the fields like medical industry, crime detection, fraudulent transaction, public safety etc. Outlier can be learnt in different fields like big data, time series data, high dimension data, biological data, uncertain data and many more. Most of the time 10% of the whole sample data set is incorrect, not accessible or missing sometimes. This paper studies and compares the popular outlier detection algorithms namely, Cluster based outlier detection, Distance based outlier detection and Density based outlier detection. Comparative study of these outlier detection techniques is performed to find out most efficient outlier detection method for calculation of the outlier.

View PDFchevron_right

References (40)

  1. Let {d M in (p, M C 1 ),. . . , d M in (p, M C l+n-1 )} be sorted in increasing order, then a lower bound on the k-distance of p, denoted as min k dist(p) will be d M in (p, M C i ) such that n 1 + ... + n i ≥ k, and n 1 + ... + n i-1 < k 2. Let {d M ax (p, M C 1 ),. . . , d M ax (p, M C l+n-1 )} be sorted in increasing order, then an upper bound on the k-distance of p, denoted as max k dist(p) will be d M ax (p, M C i ) such that n 1 + ... + n i ≥ k and n 1 + ... + n i-1 < k. The following is the micro-cluster based algorithm for mining top-n local outliers. Algorithm 3 Micro-cluster method. Input: A set of micro-clusters M C 1 , . . . , M C l , M . Output: Top-n IN F LO of D. Method:
  2. FOR each micro-cluster M C i DO
  3. Get Max/Min of k dist(p) ; // based on theorem 2
  4. IF Min k dist (p) < Mink dist (M C i ) THEN
  5. Min k dist (M C i ) = Mink dist (p);
  6. IF Max k dist (p) > Maxk dist (M C i ) THEN
  7. Max k dist (M C i ) = Maxk dist (p);
  8. FOR each micro-cluster M C i DO 9. count = |RN N k (M C i )|;
  9. IF unvisited(M C i ) THEN
  10. S = getKN N (M C i );
  11. unvisited(M C i ) = F ALSE;
  12. ELSE
  13. S = KN N (M C i );
  14. /
  15. C. Aggarwal and P. Yu: Outlier Detection for High Dimensional Data. SIGMOD 2001
  16. M. M. Breunig, H.P. Kriegel, R.T. Ng, and J.Sander: LOF: Identifying Density-based Local Outliers. SIGMOD 2000
  17. D. Chakrabarti: AutoPart: Parameter-Free Graph Partitioning and Outlier Detection. PKDD 2004
  18. Z. X. Chen, A. W. Fu, J. Tang: On Complementarity of Cluster and Outlier Detection Schemes. DaWaK 2003
  19. A. L. Chiu, A. W. Fu: Enhancements on Local Outlier Detection. IDEAS 2003
  20. M. Ester, H. P. Kriegel et al.: A Density-based Algorithm for Discovering Clusters in Large Spatial Databases. KDD 1996
  21. S. Guha, R. Rastogi, and K.Shim: Cure: An Efficient Clustering Algorithm for Large Databases. SIGMOD 1998
  22. V. Hautamki, I. Krkkinen and P. Frnti: Outlier Detection Using k-nearest Neighbour Graph, ICPR 2004.
  23. J. W. Han, M. Kamber: Data Mining: Concepts and Techniques. In Morgan Kaufmann Publishers.
  24. H. Jagadish, N. Koudas, and S. Muthukrishnan: Mining Deviants in a Time Series Database. VLDB 1999
  25. W. Jin, K. H. Tung and J. W. Han: Mining Top-n Local Outliers in Large Databases. KDD 2001
  26. E. Knorr, R. Ng: Algorithms for Mining Distance-Based Outliers in Large Datasets. VLDB 1998
  27. E. Knorr and R. Ng: Finding Intensional Knowledge of Distance-Based Outliers. VLDB 1999
  28. F. Korn and S. Muthukrishnan: Influence Sets Based on Reverse Nearest Neighbor Queries. SIGMOD 2000
  29. S. Muthukrishnan, R. Shah, J. S. Vitter: Mining Deviants in Time Series Data Streams. SSDBM 2004
  30. R. Ng and J. W. Han: Efficient and Effective Clustering Method for Spatial Data Mining. VLDB 1994
  31. S. Papadimitriou, H. Kitagawa et al. LOCI: Fast Outlier Detection Using the Local Correlation Integral. ICDE 2003
  32. S. Papadimitriou, C. Faloutsos: Cross-Outlier Detection. SSTD 2003
  33. N. Roussopoulos, S. Kelley and F. Vincent: Nearest neighbor queries. SIGMOD 1995
  34. S. Ramaswamy, R. Rastogi, K. Shim: Efficient Algorithms for Mining Outliers from Large Data Sets. SIGMOD 2000
  35. S. Shekhar, C. T. Lu, P. S. Zhang: Detecting Graph-based Spatial Outliers. KDD 2001
  36. J. Tang, Z. X. Chen et al.: Enhancing Effectiveness of Outlier Detections for Low Density Patterns. PAKDD 2002
  37. W. K. Wong, A. W. Moore et al.: Rule-Based Anomaly Pattern Detection for Detecting Disease Outbreaks. AAAI 2002
  38. M. L. Yiu, N. Mamoulis: Clustering Objects on a Spatial Network. SIGMOD 2004
  39. M. L. Yiu et al.: Aggregate Nearest Neighbor Queries in Road Networks. IEEE Trans. Knowl. Data Eng. 17(6), 2005
  40. T. Zhang et al.: BIRCH: An Efficient Data Clustering Method for Very Large Databases. SIGMOD 1996

Related papers

ARDV: A New Density Based Outlier Mining Approach 1
Govind Jha

2014

Local Outlier Factor (LOF) is an important and well known density based outliers handling algorithm, which quantifies, how much an object is outlying, in a given database. In this paper first we discuss LOF then we introduce the concept of ARDV. In LOF there is a concept of lrd (local reachability density). If in place of lrd we calculate ard (average reachability distance) and in place of LOF we calculate variance in ard (ARDV) then experimental results show that percentage of detecting correct outliers increases without increasing time complexity. Definition 2 (k-distance neighborhood of an object p represented as Nk(p)):this is the total no of object whose distance from p is not greater than the kdistance.

View PDFchevron_right
LOF: identifying density-based local outliers
Raymond Ng

ACM SIGMOD Record, 2000

For many KDD applications, such as detecting criminal activities in E-commerce, finding the rare instances or the outliers, can be more interesting than finding the common patterns. Existing work in outlier detection regards being an outlier as a binary property. In this paper, we contend that for many scenarios, it is more meaningful to assign to each object a degree of being an outlier. This degree is called the local outlier factor (LOF) of an object. It is local in that the degree depends on how isolated the object is with respect to the surrounding neighborhood. We give a detailed formal analysis showing that LOF enjoys many desirable properties. Using realworld datasets, we demonstrate that LOF can be used to find outliers which appear to be meaningful, but can otherwise not be identified with existing approaches. Finally, a careful performance evaluation of our algorithm confirms we show that our approach of finding local outliers can be practical.

View PDFchevron_right
DETECTION OF OUTLIERS BY MAKING DISTANCE-BASED METHOD
IJESRT Journal

Outlier detection in high-dimensional data presents various challenges resulting from the curse of dimensionality. A prevailing view is that distance concentration, the tendency of distances in high-dimensional data to become indiscernible, hinders the detection of outliers by making distance-based methods label all points as almost equally good outliers. In this paper, we provide evidence supporting the opinion that such a view is too simple, by demonstrating that distance-based methods can produce more contrasting outlier scores in high-dimensional settings. Furthermore, we show that high dimensionality can have a different impact, by reexamining the notion of reverse nearest neighbors in the unsupervised outlier-detection context. Namely, it was recently observed that the distribution of points' reverse-neighbor counts becomes skewed in high dimensions, resulting in the phenomenon known as hubness. We provide insight into how some points like antihubs appear very infrequently in k-NN lists of other points, and explain the connection between antihubs, outliers, and existing unsupervised outlier-detection methods. By evaluating the classic k-NN method, the angle-based technique designed for high-dimensional data, the density-based local outlier factor and influenced outlierness methods, and antihub-based methods on various synthetic and real-world data sets, we offer novel insight into the usefulness of reverse neighbor counts in unsupervised outlier detection.

View PDFchevron_right
A Mass-Based Approach for Local Outlier Detection
Anh Nguyen Hoang

IEEE Access, 2021

This paper proposes a new outlier detection approach that measures the degree of outlierness for each instance in a given dataset. The proposed model utilizes a mass-based dissimilarity measure to address the weaknesses of neighbor-based outlier models while detecting local outliers in the dataset within a variety of data point densities. In particular, it first applies a hierarchical partitioning technique to generate a set of tree-like nested structure partitions for the input dataset, and then a mass-based dissimilarity measure is defined to quantify the dissimilarity between two data instances given the generated hierarchical partition structure. After that, for each data instance, a context set is obtained by gathering the neighbors around it with the k lowest mass dissimilarities, and based on those context sets, a mass-based local outlier score model is introduced to compute the outlierness for each individual instance. The proposed approach fundamentally changes the perspective of the outlier model by using the mass-based measurement instead of the distance-based functions used in most neighbor-based methods. A comprehensive experiment conducted on both synthetic and real-world datasets demonstrates that the proposed approach is not only competitive with the existing state-of-the-art outlier detection models but is also an efficient and effective alternative for local outlier detection methods. INDEX TERMS Outlier detection, mass-based dissimilarity, unsupervised learning, knowledge discovery.

View PDFchevron_right
Reverse Nearest Neighbors in Unsupervised Distance-Based Outlier Detection
Milos Radovanovic

IEEE Transactions on Knowledge and Data Engineering, 2015

Outlier detection in high-dimensional data presents various challenges resulting from the "curse of dimensionality." A prevailing view is that distance concentration, i.e., the tendency of distances in high-dimensional data to become indiscernible, hinders the detection of outliers by making distance-based methods label all points as almost equally good outliers. In this paper we provide evidence supporting the opinion that such a view is too simple, by demonstrating that distance-based methods can produce more contrasting outlier scores in high-dimensional settings. Furthermore, we show that high dimensionality can have a different impact, by reexamining the notion of reverse nearest neighbors in the unsupervised outlier-detection context. Namely, it was recently observed that the distribution of points' reverse-neighbor counts becomes skewed in high dimensions, resulting in the phenomenon known as hubness. We provide insight into how some points (antihubs) appear very infrequently in k-NN lists of other points, and explain the connection between antihubs, outliers, and existing unsupervised outlier-detection methods. By evaluating the classic k-NN method, the angle-based technique (ABOD) designed for high-dimensional data, the density-based local outlier factor (LOF) and influenced outlierness (INFLO) methods, and antihub-based methods on various synthetic and real-world data sets, we offer novel insight into the usefulness of reverse neighbor counts in unsupervised outlier detection.

View PDFchevron_right

Related topics

  • Computer Science
  • Anomaly Detection
  • Data Mining and Knowledge Discovery
  • local outlier factor (LOF)
    • 580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved