Deepness Analysis of Learned Factors in Multilayer NMF

ArXiv, 2020

Constrained low-rank matrix approximations have been known for decades as powerful linear dimensionality reduction techniques to be able to extract the information contained in large data sets in a relevant way. However, such low-rank approaches are unable to mine complex, interleaved features that underlie hierarchical semantics. Recently, deep matrix factorization (deep MF) was introduced to deal with the extraction of several layers of features and has been shown to reach outstanding performances on unsupervised tasks. Deep MF was motivated by the success of deep learning, as it is conceptually close to some neural networks paradigms. In this paper, we present the main models, algorithms, and applications of deep MF through a comprehensive literature review. We also discuss theoretical questions and perspectives of research.

2007

The most popular algorithms for Nonnegative Matrix Factorization (NMF) belong to a class of multiplicative Lee-Seung algorithms which have usually relative low complexity but are characterized by slow-convergence and the risk of getting stuck to in local minima. In this paper, we present and compare the performance of additive algorithms based on three different variations of a projected gradient approach. Additionally, we discuss a novel multilayer approach to NMF algorithms combined with multistart initializations procedure, which in general, considerably improves the performance of all the NMF algorithms. We demonstrate that this approach (the multilayer system with projected gradient algorithms) can usually give much better performance than standard multiplicative algorithms, especially, if data are ill-conditioned, badly-scaled, and/or a number of observations is only slightly greater than a number of nonnegative hidden components. Our new implementations of NMF are demonstrated with the simulations performed for Blind Source Separation (BSS) data.

Neural Computation, 2009

Object representation in the inferior temporal cortex (IT), an area of visual cortex critical for object recognition in the primate, exhibits two prominent properties; (1) o b j e c t s a r e r e p r e s e n t e d b y t h e c o m b i n e d a c t i v i t y o f c o l u m n a r c l u s t e r s o f neurons, each cluster represents component features or parts of objects, and (2) closely related features are continuously represented along the tangential direction of individual columnar clusters. Here we propose a learning model that reflects these properties of parts-based representation and topographic organization in a unified framework. This model is based on a non-negative matrix factorization (NMF) basis-decomposition method. NMF alone provides a parts-based representation where non-negative inputs are approximated by additive combinations of non-negative basis functions. Our proposed model of topographic NMF (TNMF) incorporates neighborhood connections between NMF basis functions arranged on a topographic map and attains the topographic property without losing the parts-based property of the NMF. The TNMF represents an input by multiple activity peaks to describe diverse information whereas conventional topographic models, such as self-organizing map (SOM), represent an input by a single activity peak in a topographic map. We demonstrate the parts-based and topographic properties of the TNMF by constructing a hierarchical model for object recognition where the TNMF is at the top tier for learning high-level object features. The TNMF showed better generalization performance over NMF for a data set of continuous view change of an image, with more robustly preserving the continuity of the view change in its object representation. Comparison of the outputs of our model with actual neural responses recorded in the IT indicates that the TNMF reconstructs the neuronal responses better than the SOM, giving plausibility to the parts-based learning of the model.

Cornell University - arXiv, 2022

Various Non-negative Matrix factorization (NMF) based methods add new terms to the cost function to adapt the model to specific tasks, such as clustering, or to preserve some structural properties in the reduced space (e.g., local invariance). The added term is mainly weighted by a hyper-parameter to control the balance of the overall formula to guide the optimization process towards the objective. The result is a parameterized NMF method. However, NMF method adopts unsupervised approaches to estimate the factorizing matrices. Thus, the ability to perform prediction (e.g. classification) using the new obtained features is not guaranteed. The objective of this work is to design an evolutionary framework to learn the hyper-parameter of the parameterized NMF and estimate the factorizing matrices in a supervised way to be more suitable for classification problems. Moreover, we claim that applying NMF-based algorithms separately to different class-pairs instead of applying it once to the whole dataset improves the effectiveness of the matrix factorization process. This results in training multiple parameterized NMF algorithms with different balancing parameter values. A cross-validation combination learning framework is adopted and a Genetic Algorithm is used to identify the optimal set of hyper-parameter values. The experiments we conducted on both real and synthetic datasets demonstrated the effectiveness of the proposed approach.

In this chapter we discuss how to learn an optimal manifold presentation to regularize nonegative matrix factorization (NMF) for data representation problems. NMF, which tries to represent a nonnegative data matrix as a product of two low rank nonnegative matrices, has been a popular method for data representation due to its ability to explore the latent part-based structure of data. Recent study shows that lots of data distributions have manifold structures, and we should respect the manifold structure when the data are represented. Recently, manifold regularized NMF used a nearest neighbor graph to regulate the learning of factorization parameter matrices and has shown its advantage over traditional NMF methods for data representation problems. However, how to construct an optimal graph to present the manifold properly remains a difficult problem due to the graph model selection, noisy features, and nonlinear distributed data. In this chapter, we introduce three effective methods to solve these problems of graph construction for manifold regularized NMF. Multiple graph learning is proposed to solve the problem of graph model selection, adaptive graph learning via feature selection is proposed to solve the problem of constructing a graph from noisy features, while multi-kernel learning-based graph construction is used to solve the problem of learning a graph from nonlinearly distributed data.

ITM Web of Conferences

What matrix factorization methods do is reduce the dimensionality of the data without losing any important information. In this work, we present the Non-negative Matrix Factorization (NMF) method, focusing on its advantages concerning other methods of matrix factorization. We discuss the main optimization algorithms, used to solve the NMF problem, and their convergence. The paper also contains a comparative study between principal component analysis (PCA), independent component analysis (ICA), and NMF for dimensionality reduction using a face image database.

Neural Networks, 2003. Proceedings of the …, 2003

Discovering a representation that reflects the structure of a dataset is a first step for many inference and learning methods. This paper aims at finding a hierarchy of localized speech features that can be interpreted as parts.

We discuss Non-negative Matrix Factorization (NMF) techniques from the point of view of Intelligent Data Analysis (IDA), i.e. the intelligent application of human expertise and computational models for advanced data analysis. As IDA requires human involvement in the analysis process, the understandability of the results coming from computational models has a prominent importance. We therefore review the latest developments of NMF that try to fulfill the understandability requirement in several ways. We also describe a novel method to decompose data into user-defined — hence understandable — parts by means of a mask on the feature matrix, and show the method's effectiveness through some numerical examples.

Ecai, 2002

Properties of a novel algorithm called non-negative matrix factorization (NMF), are studied. NMF can discover substructures and can provide estimations about the presence or the absence of those, being attractive for completion of missing information. We have studied the working and learning capabilities of NMF networks. Performance was improved by adding sparse code shrinkage (SCS) algorithm to remove structureless noise. We have found that NMF performance is considerably improved by SCS noise filtering. For improving noise resistance in the learning phase, weight sparsification was studied; a sparsifying prior was applied on the NMF weight matrix. Learning capability versus noise content was measured with and without sparsifying prior. In accordance with observation made by others on independent component analysis, we have also found that weight sparsification improved learning capabilities in the presence of Gaussian noise.

INTERNATIONAL SCIENTIFIC AND PRACTICAL CONFERENCE “TECHNOLOGY IN AGRICULTURE, ENERGY AND ECOLOGY” (TAEE2022)

Network and tensor factorization are significant for the investigation of a non-negative informational index in lattice and higher request tensor structure. Such sort of factorization is known as non-negative matrix (NM) and non-negative tensor decomposition. We investigate non-negative matrix factorization (NMF) with several models in this research. Here, we look at some key models and their equations before looking at two different multiplicative algorithms for NMF. Further, investigated that comprehensive Kullback-Leibler divergence is minimized by one algorithm while the predictable least-squares error is minimized by another. Algorithm for NMF has been studied, as well as certain key theorems.