Spectral analysis of 2D outlier layout

This paper deals with a problem of identifying purely stochastic linear systems from stationary Gaussian processes with observation outliers. Reviewing [1], the spectral density function of the process under observation outliers is derived, and it is compared with the one free from outliers. It is shown that the estimates of the spectral density function is enormously affected by observation outliers on the frequency where the spectral density function is small.

IEEE Transactions on Signal Processing, 2019

Robust PCA, the problem of PCA in the presence of outliers has been extensively investigated in the last few years. Here we focus on Robust PCA in the outlier model where each column of the data matrix is either an inlier or an outlier. Most of the existing methods for this model assumes either the knowledge of the dimension of the lower dimensional subspace or the fraction of outliers in the system. However in many applications knowledge of these parameters is not available. Motivated by this we propose a parameter free outlier identification method for robust PCA which a) does not require the knowledge of outlier fraction, b) does not require the knowledge of the dimension of the underlying subspace, c) is computationally simple and fast d) can handle structured and unstructured outliers. Further, analytical guarantees are derived for outlier identification and the performance of the algorithm is compared with the existing state of the art methods in both real and synthetic data for various outlier structures.

ArXiv, 2017

Outlier detection and cluster number estimation is an important issue for clustering real data. This paper focuses on spectral clustering, a time-tested clustering method, and reveals its important properties related to outliers. The highlights of this paper are the following two mathematical observations: first, spectral clustering's intrinsic property of an outlier cluster formation, and second, the singularity of an outlier cluster with a valid cluster number. Based on these observations, we designed a function that evaluates clustering and outlier detection results. In experiments, we prepared two scenarios, face clustering in photo album and person re-identification in a camera network. We confirmed that the proposed method detects outliers and estimates the number of clusters properly in both problems. Our method outperforms state-of-the-art methods in both the 128-dimensional sparse space for face clustering and the 4,096-dimensional non-sparse space for person re-identif...

Fluctuation and Noise Letters, 2014

The aim of this paper is to study the effect of outliers on different parts of singular spectrum analysis (SSA) from both theoretical and practical points of view. The rank of the trajectory matrix, the magnitude of eigenvalues, reconstruction, and forecasting results are evaluated using simulated and real data sets. The performance of both recurrent and vector forecasting procedures are assessed in the presence of outliers. We find that the existence of outliers affect the rank of the matrix and increases the linear recurrent dimensions whilst also having a significant impact on SSA reconstruction and forecasting processes. There is also evidence to suggest that in the presence of outliers, the vector SSA forecasts are more robust in comparison to the recurrent SSA forecasts. These results indicate that the identification and removal of the outliers are mandatory to achieve optimal SSA decomposition and forecasting results.

2019

In statistics and data science, outliers are data points that differ greatly from other observations in a data set. They are important attributes of the data because they can dramatically influence patterns and relationships manifested by non-outliers. It is therefore very important to detect and adequately deal with outliers. Recently, a novel algorithm, the ROMA algorithm, has been proposed [11]. In this paper, we propose a modification of the ROMA algorithm that reduces its computational complexity from O(n 2 m) to O((n/(2 m − o(1))) 2 m) where n is the number of data points and m is the dimension of the space. And as a consequence, if log(n) < 2 m , then the improved complexity is O((n/ log(n)) 2 m).

Abstract Today PCA is very popular in various areas of pattern recognition, computer vision, and image retrieval, so its robustness to interfering data variations, eg nuisance image detail, is of a great practical value. This paper proposes to build a robust PCA by ���soft masking��� artefacts in an image database. Data variations with respect to a linear subspace of PCs are described by the mixture of Gaussian noise and uniformly distributed outliers.

2017

The curse of outlier measurements in estimation problems is a well known issue in a variety of fields. Therefore, outlier removal procedures, which enables the identification of spurious measurements within a set, have been developed for many different scenarios and applications. In this paper, we propose a statistically motivated outlier removal algorithm for time differences of arrival (TDOAs), or equivalently range differences (RD), acquired at sensor arrays. The method exploits the TDOA-space formalism and works by only knowing relative sensor positions. As the proposed method is completely independent from the application for which measurements are used, it can be reliably used to identify outliers within a set of TDOA/RD measurements in different fields (e.g. acoustic source localization, sensor synchronization, radar, remote sensing, etc.). The proposed outlier removal algorithm is validated by means of synthetic simulations and real experiments.

Malaysian Journal of Science

The existence of outlier may affect data aberrantly. However, outlier detection problem has been frequently discussed for linear data but limited on circular data. Thus, this paper discusses an outlier detection method on circular data. We focus on circular data with equal error concentration parameters where the data is studied using linear functional relationship model. In this paper, the data and the error terms are distributed with von Mises distribution. We modify the covratio statistics in which the correction factor is applied to the estimation of concentration parameter. We develop the cutoff equation based on the 5% upper percentile of the covratio statistics and the power of performance of outlier detection is examined by a Monte Carlo simulation study. The simulation result shows that the power of performance increases when the concentration and the level of contamination increase. The applicability of the proposed method is illustrated by using the wind direction data collected from the Holderness Coastline at the Humberside Coast in North Sea, United Kingdom.

The random error is following the features of normal distribution function (NDF) which those random errors deviated from the NDF&#39;s characteristics can be considered as outliers. In fact, the outliers exist inevitably in any observed parameter that is an undesirable part of the measurement&#39;s procedure due to its negative influence on the sensitivity analysis. It is therefore necessary to investigate more efficient methodologies especially for current remote sensing data processing and assimilations. In this paper, the comparisons of Baarda method as the conventional statistical methodology with the Fuzzy approach are presented to detect the outliers at the edges of two data groups: 1. The point cloud of ground-based laser scanner field experiment from one side of a wall, and 2. A group of randomly simulated distributed 3D point cloud. The results show that the Baarda method eliminates the outliers as soon as they are being found while the Fuzzy approach works critically based...

Journal of Computational and Graphical Statistics, 2007

A powerful procedure for outlier detection and robust estimation of shape and location with multivariate data in high dimension is proposed. The procedure searches for outliers in univariate projections on directions that are obtained both randomly, as in the Stahel-Donoho method, and by maximizing and minimizing the kurtosis coefficient of the projected data, as in the Peña and Prieto method. We propose modifications of both methods to improve their computational efficiency and combine them in a procedure which is affine equivariant, has a high breakdown point, is fast to compute and can be applied when the dimension is large. Its performance is illustrated with a Monte Carlo experiment and in a real dataset.