Random Projection in supervised non-stationary environments

Advances in Intelligent Systems and Computing, 2015

— An increasing number of real-world applications are associated with streaming data drawn from drifting and nonstationary distributions that change over time. These applications demand new algorithms that can learn and adapt to such changes, also known as concept drift. Proper characterization of such data with existing approaches typically requires substantial amount of labeled instances, which may be difficult, expensive, or even impractical to obtain. In this paper, we introduce compacted object sample extraction (COMPOSE), a computational geometry-based framework to learn from nonstationary streaming data, where labels are unavailable (or presented very sporadically) after initialization. We introduce the algorithm in detail, and discuss its results and performances on several synthetic and real-world data sets, which demonstrate the ability of the algorithm to learn under several different scenarios of initially labeled streaming environments. On carefully designed synthetic data sets, we compare the performance of COMPOSE against the optimal Bayes classifier, as well as the arbitrary subpopula-tion tracker algorithm, which addresses a similar environment referred to as extreme verification latency. Furthermore, using the real-world National Oceanic and Atmospheric Administration weather data set, we demonstrate that COMPOSE is competitive even with a well-established and fully supervised nonstationary learning algorithm that receives labeled data in every batch. Index Terms— Alpha shape, concept drift, nonstationary environment, semisupervised learning (SSL), verification latency.

Pattern Recognition Letters, 2012

In this text we propose a method which efficiently performs clustering of high-dimensional data. The method builds on random projection and the Kmeans algorithm. The idea is to apply K-means several times, increasing the dimensionality of the data after each convergence of K-means. We compare the proposed algorithm on four high-dimensional datasets, image, text and two synthetic, with K-means clustering using a single random projection and K-means clustering of the original high-dimensional data. Regarding time we observe that the algorithm reduces drastically the time when compared to K-means on the original high-dimensional data. Regarding mean squared error the proposed method reaches a better solution than clustering using a single random projection. More notably in the experiments performed it also reaches a better solution than clustering on the original high-dimensional data.

2009

Probabilistic Prediction Using Embedded Random Projections of High Dimensional Data. (May 2009) Richard Cable Kurwitz, B.S.; M.S., Texas A&M University Chair of Advisory Committee: Dr. Frederick R. Best The explosive growth of digital data collection and processing demands a new approach to the historical engineering methods of data correlation and model creation. A new prediction methodology based on high dimensional data has been developed. Since most high dimensional data resides on a low dimensional manifold, the new prediction methodology is one of dimensional reduction with embedding into a diffusion space that allows optimal distribution along the manifold. The resulting data manifold space is then used to produce a probability density function which uses spatial weighting to influence predictions i.e. data nearer the query have greater importance than data further away. The methodology also allows data of differing phenomenology e.g. color, shape, temperature, etc to be hand...

2003

Abstract We investigate how random projection can best be used for clustering high dimensional data. Random projection has been shown to have promising theoretical properties. In practice, however, we find that it results in highly unstable clustering performance. Our solution is to use random projection in a cluster ensemble approach.

Many tasks (e.g., clustering) in machine learning only require the l α distances instead of the original data. For dimension reductions in the l α norm (0 < α ≤ 2), the method of stable random projections can efficiently compute the l α distances in massive datasets (e.g., the Web or massive data streams) in one pass of the data. The estimation task for stable random projections has been an interesting topic. We propose a simple estimator based on the fractional power of the samples (projected data), which is surprisingly near-optimal in terms of the asymptotic variance. In fact, it achieves the Cramér-Rao bound when α = 2 and α = 0+. This new result will be useful when applying stable random projections to distancebased clustering, classifications, kernels, massive data streams etc.

Information Sciences, 2006

Algorithms on streaming data have attracted increasing attention in the past decade. Among them, dimensionality reduction algorithms are greatly interesting due to the desirability of real tasks. Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA) are two of the most widely used dimensionality reduction approaches. However, PCA is not optimal for general classification problems because it is unsupervised and ignores valuable label information for classification. On the other hand, the performance of LDA is degraded when encountering limited available low-dimensional spaces and singularity problem. Recently, Maximum Margin Criterion (MMC) was 0020-0255/$ -see front matter Ó proposed to overcome the shortcomings of PCA and LDA. Nevertheless, the original MMC algorithm could not satisfy the streaming data model to handle large-scale high-dimensional data set. Thus an effective, efficient and scalable approach is needed. In this paper, we propose a supervised incremental dimensionality reduction algorithm and its extension to infer adaptive low-dimensional spaces by optimizing the maximum margin criterion. Experimental results on a synthetic dataset and real datasets demonstrate the superior performance of our proposed algorithm on streaming data.

Expert Systems with Applications, 2024

This paper introduces a novel hybrid random projection method (HRP) that effectively combines the strengths of normal random projection (NRP) and plus-minus one random projection (PMRP). By incorporating a blending parameter, HRP optimises the contributions of the NRP and PMRP, reducing the dimensions of the data while ensuring that the structure of the data is preserved. Conventional methods, such as NRP and PMRP, are recognised for their benefits; however, they also have limitations. NRP is generally accurate but can encounter difficulties with certain data types or noise. The PMRP is simple, but occasionally fails to capture complex relationships within the data. The performance of HRP is investigated through comprehensive evaluations, including simulations and real-world data analyses from key machine learning datasets, such as period change, toxicity, MNIST, and human activity recognition (HAR). We examine various factors such as sample size, dimensions of the original and reduced data, and sparsity. Distance distortion is the primary metric utilised to measure performance, and indicates how well the dataset structure is maintained after dimension reduction. In every test, HRP consistently demonstrates the least distance distortion, performing superiorly to the NRP and PMRP. This is true for a range of scenarios and datasets. HRP represents a significant advancement in dimension reduction techniques. Its ability to minimise information loss during the reduction process demonstrates its potential as a powerful tool for managing complex high-dimensional data in practical applications. This renders HRP an important development in the fields of machine learning, intelligent systems, and artificial intelligence, offering improved efficiency and precision in data analyses.

2012

The primary focus of machine learning has traditionally been on learning from data assumed to be sufficient and representative of the underlying fixed, yet unknown, distribution. Such restrictions on the problem domain paved the way for development of elegant algorithms with theoretically provable performance guarantees. As is often the case, however, real world problems rarely fit neatly into such restricted models. For instance class distributions are often skewed, resulting in the "class imbalance" problem. Data drawn from non-stationary distributions is also common in real world applications, resulting in the "concept drift" or "non-stationary learning" problem which is often associated with streaming data scenarios. Recently, these problems have independently experienced increased research attention, however the combined problem of addressing all of the above mentioned issues has enjoyed relatively little research. If the ultimate goal of intelligent machine learning algorithms is to be able to address a wide spectrum of real world scenarios, then the need for a general framework for learning from, and adapting to, a non-stationary environment that may introduce imbalanced data can be hardly overstated. In this paper, we first present an overview of each of these challenging areas, followed by a comprehensive review of recent research for developing such a general framework.

2012

LP-type Problems Definition [MS03] An abstract LP-type problem is a pair (H,w) where: H a finite set of constraints. w : H2 → R ∪ (−∞,∞) an objective function to be minimized which satisfies, for any h ∈ H and any F ⊆ G ⊆ H: Monotonicity: w(F ) 6 w(G) 6 w(H). Locality: If w(F ) = w(G) = w(F ∪ h) then w(F ) = w(G ∪ h). Interpretation: w(G) is the minimum value of a solution satisfying all constraints on G. R.J.Durrant & A.Kaban (U.Birmingham) RP for Machine Learning & Data Mining ECML-PKDD 2012 67 / 123 Basis and Combinatorial Dimension Definitions: Basis, Combinatorial Dimension L = (H,w) abstract LP-type problem then: A basis for F ⊆ H is a minimal set of constraints B ⊆ F such that w(B) = w(F ). The combinatorial dimension of L is the size of the largest basis. Combinatorial dimension examples for problems in Rd : Smallest enclosing ball, d + 1 Linear program, d + 1 Distance between hyperplanes, d + 2 R.J.Durrant & A.Kaban (U.Birmingham) RP for Machine Learning & Data Mining ECML-...