Learning the sparse representation for classification

Linear approaches to the problem of unsupervised data dimensionality reduction consist in finding a suitable set of factors, which is usually called dictionary, on the basis of which data can be represented as a linear combination of the dictionary elements. In recent years there have been relevant efforts for searching data representation which are based on sparse dictionary elements or a sparse linear combination of the dictionary elements. Here we investigate the possibility to combine the advantages of both sparse dictionary elements and sparse linear combination. Notably, we also impose a structure on the dictionary elements. We compare our algorithm with two other different approaches presented in literature which impose either sparse structured dictionary elements or sparse linear combination. These (preliminary) results suggests that our approach presents some promising advantages, in particular a greater possibility of interpreting the data representation.

IEEE Transactions on Image Processing, 2014

In complex visual recognition tasks it is typical to adopt multiple descriptors, that describe different aspects of the images, for obtaining an improved recognition performance. Descriptors that have diverse forms can be fused into a unified feature space in a principled manner using kernel methods. Sparse models that generalize well to the test data can be learned in the unified kernel space, and appropriate constraints can be incorporated for application in supervised and unsupervised learning. In this paper, we propose to perform sparse coding and dictionary learning in the multiple kernel space, where the weights of the ensemble kernel are tuned based on graph-embedding principles such that class discrimination is maximized. In our proposed algorithm, dictionaries are inferred using multiple levels of 1−D subspace clustering in the kernel space, and the sparse codes are obtained using a simple levelwise pursuit scheme. Empirical results for object recognition and image clustering show that our algorithm outperforms existing sparse coding based approaches, and compares favorably to other state-of-the-art methods.

2011

Learning sparse representations on data adaptive dictionaries is a state-of-the-art method for modeling data. But when the dictionary is large and the data dimension is high, it is a computationally challenging problem. We explore three aspects of the problem. First, we derive new, greatly improved screening tests that quickly identify codewords that are guaranteed to have zero weights. Second, we study the properties of random projections in the context of learning sparse representations. Finally, we develop a hierarchical framework that uses incremental random projections and screening to learn, in small stages, a hierarchically structured dictionary for sparse representations. Empirical results show that our framework can learn informative hierarchical sparse representations more efficiently.

Neurocomputing, 2014

In the recent years, non-negative matrix factorization and sparse representation models have been successfully applied in high-throughput biological data analysis due to its interpretability and robustness to noise. In this paper, we propose a unified matrix factorization model, coined versatile sparse matrix factorization (VSMF) model, for biological data analysis. We discuss the modelling, optimization, and applications of VSMF. We show that many well-known sparse matrix factorization models are specific cases of our VSMF. Through tuning parameters, sparsity, smoothness, and non-negativity can be easily controlled in VSMF. Our computational experiments for feature extraction, feature selection, and clustering corroborate the advantages of VSMF.

IEEE Access

Sparse representation (SR) and dictionary learning (DL) have been extensively used for feature encoding, aiming to extract the latent classification-friendly feature of observed data. Existing methods use sparsity penalty and learned dictionary to enhance discriminative capability of sparse codes. However, training dictionary for SR is time consuming and the resulted discriminative capability is limited. Rather than learning dictionary, we propose to employ the dictionary at hand, e.g., the training set as the classspecific synthesis dictionary to pursue an ideal discriminative property of SR of the training samples: each data can be represented only by data-in-class. In addition to the discriminative property, we also introduce a smoothing term to enforce the representation vectors to be uniform within class. The discriminative property helps to separate the data from different classes while the smoothing term tends to group the data from the same class and further strengthen the separation. The SRs are used as new features to train a sparse encoder and a classifier. Once the sparse encoder and the classifier are learnt, the test stage is very simple and highly efficient. Specifically, the label of a test sample can be easily computed by multiplying the test sample with the sparse encoder and the classifier. We call our method Classification-Friendly Sparse Encoder and Classifier Learning (CF-SECL). Extensive experiments show that our method outperforms some state-of the-art model-based methods.

2012

Sparse representations account for most or all of the information of a signal by a linear combination of a few elementary signals called atoms, and have increasingly become recognized as providing high performance for applications as diverse as noise reduction, compression, inpainting, compressive sensing, pattern classification, and blind source separation. In this dissertation, we learn the sparse representations of high-dimensional signals for various learning and vision tasks, including image classification, single image super-resolution, compressive sensing, and graph learning. Based on the bag-of-features (BoF) image representation in a spatial pyramid, we first transform each local image descriptor into a sparse representation, and then these sparse representations are summarized into a fixed-length feature vector over different spatial locations across different spatial scales by max pooling. The proposed generic image feature representation properly handles the large in-class variance problem in image classification, and experiments on object recognition, scene classification, face recognition, gender recognition, and handwritten digit recognition all lead to state-of-the-art performances on the benchmark datasets. We cast the image super-resolution problem as one of recovering a highresolution image patch for each low-resolution image patch based on recent sparse signal recovery theories, which state that, under mild conditions, a high-resolution signal can be recovered from its low-resolution version if the signal has a sparse representation in terms of some dictionary. We jointly learn the dictionaries for high-and low-resolution image patches and enforce them to have common sparse representations for better recovery. Furthermore, we employ image features and enforce patch overlapping constraints to improve prediction accuracy. Experiments show that the algorithm leads to surprisingly good results. Graph construction is critical for those graph-orientated algorithms designed for the purposes of data clustering, subspace learning, and semi-supervised learning. We model the graph construction problem, including neighbor selection and First, I would like to express profound gratitude to my adviser Professor Thomas Huang for all the guidance, advice, and support he has given me during my Ph.D. study at UIUC. For the last five and half years, I have been inspired by his vision and passion to research, his attention and curiosity to details, his dedication to the profession, his intense commitment to his work, and his humble and respectful personality. During this most important period in my career, I thoroughly enjoyed working with him, and what I have learned from him will benefit me for my whole life. I also would like to give my thanks to Professor Yi Ma for his suggestions and support for the last several years. I was deeply impressed and inspired by his passion and insights into research. I also feel very lucky to have had the chance to work with Dr. Kai Yu as a summer intern at NEC Laboratories America. From him, I learned not only techniques and skills but also his precious research insights. I am also very thankful to Professor Narendra Ahuja and Professor Zhi-Pei Liang for their advice and comments. I am fortunate to have them as my thesis committee members. During my Ph.D. study over the last five years, it has been a great honor for me to collaborate with Hasegawa-Johnson at UIUC, Professor Shuicheng Yan at

IEEE Signal Processing Letters, 2000

A dictionary learning problem is a matrix factorization in which the goal is to factorize a training data matrix, , as the product of a dictionary, , and a sparse coefficient matrix, , as follows, . Current dictionary learning algorithms minimize the representation error subject to a constraint on (usually having unit column-norms) and sparseness of . The resulting problem is not convex with respect to the pair . In this letter, we derive a first order series expansion formula for the factorization, . The resulting objective function is jointly convex with respect to and . We simply solve the resulting problem using alternating minimization and apply some of the previously suggested algorithms onto our new problem. Simulation results on recovery of a known dictionary and dictionary learning for natural image patches show that our new problem considerably improves performance with a little additional computational load.

IEEE Transactions on Signal Processing - TSP, 2005

and is a full-rank matrix, an infinite number of solutions are available for the representation problem, hence constraints on the solution must be set. The solution with the fewest number of nonzero coefficients is certainly an appealing representation. This sparsest representation is the solution of either

Proceedings of the National Academy of Sciences, 2003

Given a dictionary D ‫؍‬ {d ᠪ k} of vectors d ᠪ k, we seek to represent a signal S ᠪ as a linear combination S ᠪ ‫؍‬ ͚ k ␥(k)d ᠪ k, with scalar coefficients ␥(k). In particular, we aim for the sparsest representation possible. In general, this requires a combinatorial optimization process. Previous work considered the special case where D is an overcomplete system consisting of exactly two orthobases and has shown that, under a condition of mutual incoherence of the two bases, and assuming that S ᠪ has a sufficiently sparse representation, this representation is unique and can be found by solving a convex optimization problem: specifically, minimizing the ഞ 1 norm of the coefficients ᠪ ␥. In this article, we obtain parallel results in a more general setting, where the dictionary D can arise from two or several bases, frames, or even less structured systems. We sketch three applications: separating linear features from planar ones in 3D data, noncooperative multiuser encoding, and identification of over-complete independent component models.

2012 11th International Conference on Machine Learning and Applications, 2012

Sparse representation (SR) has been being applied as a state-of-the-art machine learning approach. Sparse representation classification (SRC1) approaches based on 1 norm regularization and non-negative-least-squares (NNLS) classification approach based on non-negativity have been proposed to be powerful and robust. However, these approaches are extremely slow when the size of training samples is very large, because both of them use the whole training set as dictionary. In this paper, we briefly survey the existing SR techniques for classification, and then propose a fast approach which uses nonnegative matrix factorization as supervised dictionary learning method and NNLS as non-negative sparse coding method. Experiment shows that our approach can obtain comparable accuracy with the benchmark approaches and can dramatically speed up the computation particularly in the case of large sample size and many classes.