Academia.eduAcademia.edu

Outline

Title
Abstract
Introduction
Multigrid Methods
Computational Results
Results
Unstructured Data
References
All Topics
Mathematics
Applied Mathematics

A multilevel approach for nonnegative matrix factorization

Profile image of Nicolas GillisNicolas Gillis

2012, Journal of Computational and Applied Mathematics

https://doi.org/10.1016/J.CAM.2011.10.002
visibility

…

description

22 pages

link

1 file

Abstract

Nonnegative Matrix Factorization (NMF) is the problem of approximating a nonnegative matrix with the product of two low-rank nonnegative matrices and has been shown to be particularly useful in many applications, e.g., in text mining, image processing, computational biology, etc. In this paper, we explain how algorithms for NMF can be embedded into the framework of multilevel methods in order to accelerate their convergence. This technique can be applied in situations where data admit a good approximate representation in a lower dimensional space through linear transformations preserving nonnegativity. A simple multilevel strategy is described and is experimentally shown to speed up significantly three popular NMF algorithms (alternating nonnegative least squares, multiplicative updates and hierarchical alternating least squares) on several standard image datasets.

Related papers

Additive Update Algorithm for Nonnegative Matrix Factorization
At Pham Van

Nonnegative matrix factorization (NMF) is an emerging technique with a wide spectrum of potential applications in data analysis. Mathematically, NMF can be formulated as a minimization problem with nonnegative constraints. This problem is currently attracting much attention from researchers for theoretical reasons and for potential applications. Currently, the most popular approach to solve NMF is the multiplicative update algorithm proposed by D.D. Lee and H.S. Seung. In this paper, we propose an additive update algorithm, that has faster computational speed than the algorithm of D.D. Lee and H.S. Seung.

View PDFchevron_right
Statistical optimization of non-negative matrix factorization
Max Welling

2011

Abstract Non-Negative Matrix Factorization (NMF) is a dimensionality reduction method that has been shown to be very useful for a variety of tasks in machine learning and data mining. One of the fastest algorithms for NMF is the Block Principal Pivoting method (BPP) of [6], which follows a block coordinate descent approach. The optimization in each iteration involves solving a large number of expensive least squares problems.

View PDFchevron_right
Nonnegative matrix factorization via rank-one downdate
Ali Ghodsi

Proceedings of the 25th international conference on Machine learning - ICML '08, 2008

Nonnegative matrix factorization (NMF) was popularized as a tool for data mining by Lee and Seung in 1999. NMF attempts to approximate a matrix with nonnegative entries by a product of two low-rank matrices, also with nonnegative entries. We propose an algorithm called rank-one downdate (R1D) for computing an NMF that is partly motivated by the singular value decomposition. This algorithm computes the dominant singular values and vectors of adaptively determined submatrices of a matrix. On each iteration, R1D extracts a rank-one submatrix from the original matrix according to an objective function. We establish a theoretical result that maximizing this objective function corresponds to correctly classifying articles in a nearly separable corpus. We also provide computational experiments showing the success of this method in identifying features in realistic datasets. The method is also much faster than other NMF routines.

View PDFchevron_right
Improving non-negative matrix factorization via ranking its bases
Ahmed Elgammal

2014 IEEE International Conference on Image Processing (ICIP), 2014

As a considerable technique in image processing and computer vision, Nonnegative Matrix Factorization (NMF) generates its bases by iteratively multiplicative update with two initial random nonnegative matrices W and H, that leads to the randomness of the bases selection. For this reason, the potentials of NMF algorithms are not completely exploited. To address this issue, we present a novel framework which uses the feature selection techniques to evaluate and rank the bases of the NMF algorithms to enhance the NMF algorithms. We adopted the well known Fisher criterion and Least Reconstruction Error criterion, which is proposed by us, as two instances to show how that works successfully under our framework. Moreover, in order to avoid the hard combinatorial optimization issue in ranking procedure, a de-correlation constraint can be optionally imposed to the NMF algorithms for giving a better approximation to the global optimum of the NMF projections. We evaluate our works in face recognition, object recognition and image reconstruction on ORL and ETH-80 databases and the results demonstrate the enhancement of the state-of-the-art NMF under our framework.

View PDFchevron_right
Algorithms for orthogonal nonnegative matrix factorization
Seungjin Choi

2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence), 2008

Nonnegative matrix factorization (NMF) is a widely-used method for multivariate analysis of nonnegative data, the goal of which is decompose a data matrix into a basis matrix and an encoding variable matrix with all of these matrices allowed to have only nonnegative elements. In this paper we present simple algorithms for orthogonal NMF, where orthogonality constraints are imposed on basis matrix or encoding matrix. We develop multiplicative updates directly from the true gradient (natural gradient) in Stiefel manifold, whereas existing algorithms consider additive orthogonality constraints. Numerical experiments on face image data for a image representation task show that our orthogonal NMF algorithm preserves the orthogonality, while the goodness-offit (GOF) is minimized. We also apply our orthogonal NMF to a clustering task, showing that it works better than the original NMF, which is confirmed by experiments on several UCI repository data sets.

View PDFchevron_right
libNMF -- A Library for Nonnegative Matrix Factorization
Wilfried Gansterer

Computing and Informatics, 2012

We present libNMF-a computationally efficient high performance library for computing nonnegative matrix factorizations (NMF) written in C. Various algorithms and algorithmic variants for computing NMF are supported. libNMF is based on external routines from Blas (Basic Linear Algebra Subprograms), Lapack (Linear Algebra package) and Arpack, which provide efficient building blocks for performing central vector and matrix operations. Since modern Blas implementations support multi-threading, libNMF can exploit the potential of multi-core architectures. In this paper, the basic NMF algorithms contained in libNMF and existing implementations found in the literature are briefly reviewed. Then, libNMF is evaluated in terms of computational efficiency and numerical accuracy and compared with the best existing codes available. libNMF is publicly available at http://rlcta.univie.ac.at/software.

View PDFchevron_right
Fast local algorithms for large scale nonnegative matrix and tensor factorizations
Anh Huy

2009

Nonnegative matrix factorization (NMF) and its extensions such as Nonnegative Tensor Factorization (NTF) have become prominent techniques for blind sources separation (BSS), analysis of image databases, data mining and other information retrieval and clustering applications. In this paper we propose a family of efficient algorithms for NMF/NTF, as well as sparse nonnegative coding and representation, that has many potential applications in computational neuroscience, multisensory processing, compressed sensing and multidimensional data analysis. We have developed a class of optimized local algorithms which are referred to as Hierarchical Alternating Least Squares (HALS) algorithms. For these purposes, we have performed sequential constrained minimization on a set of squared Euclidean distances. We then extend this approach to robust cost functions using the Alpha and Beta divergences and derive flexible update rules. Our algorithms are locally stable and work well for NMF-based blind source separation (BSS) not only for the over-determined case but also for an under-determined (over-complete) case (i.e., for a system which has less sensors than sources) if data are sufficiently sparse. The NMF learning rules are extended and generalized for N-th order nonnegative tensor factorization (NTF). Moreover, these algorithms can be tuned to different noise statistics by adjusting a single parameter. Extensive experimental results confirm the accuracy and computational performance of the developed algorithms, especially, with usage of multi-layer hierarchical NMF approach [3].

View PDFchevron_right
Nonnegative matrix factorization and its applications in pattern recognition
Andrzej Cichocki

Chinese Science Bulletin, 2006

Matrix factorization is an effective tool for large-scale data processing and analysis. Nonnegative matrix factorization (NMF) method, which decomposes the nonnegative matrix into two nonnegative factor matrices, provides a new way for matrix factorization. NMF is significant in intelligent information processing and pattern recognition. This paper firstly introduces the basic idea of NMF and some new relevant methods. Then we discuss the loss functions and relevant algorithms of NMF in the framework of probabilistic models based on our researches, and the relationship between NMF and information processing of perceptual process. Finally, we make use of NMF to deal with some practical questions of pattern recognition and point out some open problems for NMF.

View PDFchevron_right
Multilayer nonnegative matrix factorization using projected gradient approaches
Rafal Zdunek

2007

The most popular algorithms for Nonnegative Matrix Factorization (NMF) belong to a class of multiplicative Lee-Seung algorithms which have usually relative low complexity but are characterized by slow-convergence and the risk of getting stuck to in local minima. In this paper, we present and compare the performance of additive algorithms based on three different variations of a projected gradient approach. Additionally, we discuss a novel multilayer approach to NMF algorithms combined with multistart initializations procedure, which in general, considerably improves the performance of all the NMF algorithms. We demonstrate that this approach (the multilayer system with projected gradient algorithms) can usually give much better performance than standard multiplicative algorithms, especially, if data are ill-conditioned, badly-scaled, and/or a number of observations is only slightly greater than a number of nonnegative hidden components. Our new implementations of NMF are demonstrated with the simulations performed for Blind Source Separation (BSS) data.

View PDFchevron_right
Nonnegative Matrix Factorization with Earth Mover's Distance metric
Michael Lindenbaum

2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009

Nonnegative Matrix Factorization (NMF) approximates a given data matrix as a product of two low rank nonnegative matrices, usually by minimizing the L 2 or the KL distance between the data matrix and the matrix product. This factorization was shown to be useful for several important computer vision applications. We propose here a new NMF algorithm that minimizes the Earth Mover's Distance (EMD) error between the data and the matrix product. We propose an iterative NMF algorithm (EMD NMF) and prove its convergence. The algorithm is based on linear programming. We discuss the numerical difficulties of the EMD NMF and propose an efficient approximation. Naturally, the matrices obtained with EMD NMF are different from those obtained with L 2 NMF. We discuss these differences in the context of two challenging computer vision tasks-texture classification and face recognition-and demonstrate the advantages of the proposed method.

View PDFchevron_right

References (47)

  1. M. Berry, M. Browne, A. Langville, P. Pauca, and R. Plemmons, Algorithms and Applications for Approximate Nonnegative Matrix Factorization, Computational Statistics and Data Analysis, 52 (2007), pp. 155-173.
  2. C. Boutsidis and E. Gallopoulos, SVD based initialization: A head start for nonnegative matrix factorization, Journal of Pattern Recognition, 41 (2008), pp. 1350-1362.
  3. J. H. Bramble, Multigrid methods, Number 294 Pitman Research Notes in Mathematic Series. Longman Scientific & Technical, UK, 1995.
  4. A. Brandt, Guide to multigrid development, W. Hackbusch and U. Trottenberg, eds., Multigrid Methods, Lecture Notes in Mathematics, Springer, 960 (1982), pp. 220-312.
  5. W. L. Briggs, A Multigrid Tutorial, SIAM, Philadelphia, 1987.
  6. D. Chen and R. Plemmons, Nonnegativity Constraints in Numerical Analysis, in A. Bultheel and R. Cools (Eds.), Symposium on the Birth of Numerical Analysis, World Scientific Press., 2009.
  7. A. Cichocki, S. Amari, R. Zdunek, and A. Phan, Non-negative Matrix and Tensor Fac- torizations: Applications to Exploratory Multi-way Data Analysis and Blind Source Separation, Wiley-Blackwell, 2009.
  8. C. Cichocki and A.-H. Phan, Fast local algorithms for large scale Nonnegative Matrix and Tensor Factorizations, IEICE Transactions on Fundamentals of Electronics, Vol. E92-A No.3 (2009), pp. 708-721.
  9. C. Cichocki, R. Zdunek, and S. Amari, Non-negative Matrix Factorization with Quasi- Newton Optimization, in Lecture Notes in Artificial Intelligence, Springer, vol. 4029, 2006, pp. 870-879.
  10. Hierarchical ALS Algorithms for Nonnegative Matrix and 3D Tensor Factorization, in Lecture Notes in Computer Science, Vol. 4666, Springer, pp. 169-176, 2007.
  11. Nonnegative Matrix and Tensor Factorization, IEEE Signal Processing Magazine, (2008), pp. 142-145.
  12. J. Curry, A. Dougherty, and S. Wild, Improving non-negative matrix factorizations through structured initialization, Journal of Pattern Recognition, 37(11) (2004), pp. 2217-2232.
  13. M. E. Daube-Witherspoon and G. Muehllehner, An iterative image space reconstruction algorithm suitable for volume ect, IEEE Trans. Med. Imaging, 5 (1986), pp. 61-66.
  14. I. Dhillon, D. Kim, and S. Sra, Fast Newton-type Methods for the Least Squares Nonnegative Matrix Approximation problem, in Proc. of SIAM Conf. on Data Mining, 2007.
  15. N. Gillis and F. Glineur, Nonnegative Factorization and The Maximum Edge Biclique Prob- lem. CORE Discussion paper 2008/64, 2008.
  16. Nonnegative Matrix Factorization and Underapproximation. Communication at 9th Inter- national Symposium on Iterative Methods in Scientific Computing, Lille, France, 2008.
  17. S. Gratton, A. Sartenaer, and P. Toint, On Recursive Multiscale Trust-Region Algorithms for Unconstrained Minimization, in Oberwolfach Reports: Optimization and Applications.
  18. J. Han, L. Han, M. Neumann, and U. Prasad, On the rate of convergence of the image space reconstruction algorithm, Operators and Matrices, 3(1) (2009), pp. 41-58.
  19. N.-D. Ho, P. Van Dooren, and V. Blondel, Descent methods for nonnegative matrix factor- ization, In: Numerical Linear Algebra in Signals, Systems and Control, Springer Verlag, (2008).
  20. H. Kim and H. Park, Non-negative Matrix Factorization Based on Alternating Non-negativity Constrained Least Squares and Active Set Method, SIAM J. Matrix Anal. Appl., 30(2) (2008), pp. 713-730.
  21. C. Lawson and R. Hanson, Solving Least Squares Problems, Prentice-Hall, 1974.
  22. D. Lee and H. Seung, Learning the Parts of Objects by Nonnegative Matrix Factorization, Nature, 401 (1999), pp. 788-791.
  23. Algorithms for Non-negative Matrix Factorization, In Advances in Neural Information Pro- cessing, 13 (2001).
  24. C.-J. Lin, On the Convergence of Multiplicative Update Algorithms for Nonnegative Matrix Fac- torization, in IEEE Transactions on Neural Networks, 2007.
  25. Projected Gradient Methods for Nonnegative Matrix Factorization, Neural Computation, 19 (2007), pp. 2756-2779. MIT press.
  26. P. Paatero and U. Tapper, Positive matrix factorization: a non-negative factor model with optimal utilization of error estimates of data values, Environmetrics, 5 (1994), pp. 111-126.
  27. P. Pauca, J. Piper, and R. Plemmons, Nonnegative matrix factorization for spectral data analysis , Linear Algebra and its Applications, 406(1) (2006), pp. 29-47.
  28. S. Sakellaridi, H.-r. Fang, and Y. Saad, Graph-based Multilevel Dimensionality Reduction with Applications to Eigenfaces and Latent Semantic Indexing. preprint, 2009.
  29. D. Terzopoulos, Image Analysis Using Multigrid Relaxation Methods, J. Math. Phys., PAMI- 8(2) (1986), pp. 129-139.
  30. U. Trottenberg, C. Oosterlee, and A. Schüller, Multigrid, Elsevier Academic Press, London, 2001.
  31. M. Van Benthem and M. Keenan, Fast algorithm for the solution of large-scale non-negativity constrained least squares problems, J. Chemometrics, 18 (2004), pp. 441-450.
  32. S. A. Vavasis, On the complexity of nonnegative matrix factorization, SIAM Journal on Opti- mization, 20 (2009), pp. 1364-1377.
  33. Q. Zhang, H. Wang, R. Plemmons, and P. Pauca, Tensor methods for hyperspectral data analysis: a space object material identification study, J. Optical Soc. Amer. A, 25(12) (2008), pp. 3001-3012. Recent titles CORE Discussion Papers 2010/6. Marc FLEURBAEY, Stéphane LUCHINI, Christophe MULLER and Erik SCHOKKAERT. Equivalent income and the economic evaluation of health care. 2010/7. Elena IÑARRA, Conchi LARREA and Elena MOLIS. The stability of the roommate problem revisited. 2010/8. Philippe CHEVALIER, Isabelle THOMAS and David GERAETS, Els GOETGHEBEUR, Olivier JANSSENS, Dominique PEETERS and Frank PLASTRIA. Locating fire-stations: an integrated approach for Belgium. 2010/9. Jean-Charles LANGE and Pierre SEMAL. Design of a network of reusable logistic containers. 2010/10. Hiroshi UNO. Nested potentials and robust equilibria. 2010/11. Elena MOLIS and Róbert F. VESZTEG. Experimental results on the roommate problem. 2010/12. Koen DECANCQ. Copula-based orderings of multivariate dependence. 2010/13. Tom TRUYTS. Signaling and indirect taxation. 2010/14. Asel ISAKOVA. Currency substitution in the economies of Central Asia: How much does it cost? 2010/15. Emanuele FORLANI. Irish firms' productivity and imported inputs. 2010/16. Thierry BRECHET, Carmen CAMACHO and Vladimir M. VELIOV. Model predictive control, the economy, and the issue of global warming. 2010/17. Thierry BRECHET, Tsvetomir TSACHEV and Vladimir M. VELIOV. Markets for emission permits with free endowment: a vintage capital analysis. 2010/18. Pierre M. PICARD and Patrice PIERETTI. Bank secrecy, illicit money and offshore financial centers. 2010/19. Tanguy ISAAC. When frictions favour information revelation. 2010/20. Jeroen V.K. ROMBOUTS and Lars STENTOFT. Multivariate option pricing with time varying volatility and correlations. 2010/21. Yassine LEFOUILI and Catherine ROUX. Leniency programs for multimarket firms: The effect of Amnesty Plus on cartel formation. 2010/22. P. Jean-Jacques HERINGS, Ana MAULEON and Vincent VANNETELBOSCH. Coalition formation among farsighted agents. 2010/23. Pierre PESTIEAU and Grégory PONTHIERE. Long term care insurance puzzle. 2010/24. Elena DEL REY and Miguel Angel LOPEZ-GARCIA. On welfare criteria and optimality in an endogenous growth model. 2010/25. Sébastien LAURENT, Jeroen V.K. ROMBOUTS and Francesco VIOLANTE. On the forecasting accuracy of multivariate GARCH models. 2010/26. Pierre DEHEZ. Cooperative provision of indivisible public goods. 2010/27. Olivier DURAND-LASSERVE, Axel PIERRU and Yves SMEERS. Uncertain long-run emissions targets, CO 2 price and global energy transition: a general equilibrium approach. 2010/28. Andreas EHRENMANN and Yves SMEERS. Stochastic equilibrium models for generation capacity expansion. 2010/29. Olivier DEVOLDER, François GLINEUR and Yu. NESTEROV. Solving infinite-dimensional optimization problems by polynomial approximation. 2010/30. Helmuth CREMER and Pierre PESTIEAU. The economics of wealth transfer tax. 2010/31. Thierry BRECHET and Sylvette LY. Technological greening, eco-efficiency, and no-regret strategy. 2010/32. Axel GAUTIER and Dimitri PAOLINI. Universal service financing in competitive postal markets: one size does not fit all. 2010/33. Daria ONORI. Competition and growth: reinterpreting their relationship. 2010/34. Olivier DEVOLDER, François GLINEUR and Yu. NESTEROV. Double smoothing technique for infinite-dimensional optimization problems with applications to optimal control. 2010/35. Jean-Jacques DETHIER, Pierre PESTIEAU and Rabia ALI. The impact of a minimum pension on old age poverty and its budgetary cost. Evidence from Latin America. Recent titles CORE Discussion Papers -continued 2010/36. Stéphane ZUBER. Justifying social discounting: the rank-discounting utilitarian approach. 2010/37. Marc FLEURBAEY, Thibault GAJDOS and Stéphane ZUBER. Social rationality, separability, and equity under uncertainty. 2010/38. Helmuth CREMER and Pierre PESTIEAU. Myopia, redistribution and pensions. 2010/39. Giacomo SBRANA and Andrea SILVESTRINI. Aggregation of exponential smoothing processes with an application to portfolio risk evaluation. 2010/40. Jean-François CARPANTIER. Commodities inventory effect. 2010/41. Pierre PESTIEAU and Maria RACIONERO. Tagging with leisure needs. 2010/42. Knud J. MUNK. The optimal commodity tax system as a compromise between two objectives. 2010/43. Marie-Louise LEROUX and Gregory PONTHIERE. Utilitarianism and unequal longevities: A remedy? 2010/44. Michel DENUIT, Louis EECKHOUDT, Ilia TSETLIN and Robert L. WINKLER. Multivariate concave and convex stochastic dominance. 2010/45. Rüdiger STEPHAN. An extension of disjunctive programming and its impact for compact tree formulations. 2010/46. Jorge MANZI, Ernesto SAN MARTIN and Sébastien VAN BELLEGEM. School system evaluation by value-added analysis under endogeneity. 2010/47. Nicolas GILLIS and François GLINEUR. A multilevel approach for nonnegative matrix factorization.
  34. Books J. GABSZEWICZ (ed.) (2006), La différenciation des produits. Paris, La découverte.
  35. L. BAUWENS, W. POHLMEIER and D. VEREDAS (eds.) (2008), High frequency financial econometrics: recent developments. Heidelberg, Physica-Verlag.
  36. P. VAN HENTENRYCKE and L. WOLSEY (eds.) (2007), Integration of AI and OR techniques in constraint programming for combinatorial optimization problems. Berlin, Springer.
  37. P-P. COMBES, Th. MAYER and J-F. THISSE (eds.) (2008), Economic geography: the integration of regions and nations. Princeton, Princeton University Press.
  38. J. HINDRIKS (ed.) (2008), Au-delà de Copernic: de la confusion au consensus ? Brussels, Academic and Scientific Publishers.
  39. J-M. HURIOT and J-F. THISSE (eds) (2009), Economics of cities. Cambridge, Cambridge University Press.
  40. P. BELLEFLAMME and M. PEITZ (eds) (2010), Industrial organization: markets and strategies. Cambridge University Press.
  41. M. JUNGER, Th. LIEBLING, D. NADDEF, G. NEMHAUSER, W. PULLEYBLANK, G. REINELT, G. RINALDI and L. WOLSEY (eds) (2010), 50 years of integer programming, 1958-2008: from the early years to the state-of-the-art. Berlin Springer. CORE Lecture Series C. GOURIÉROUX and A. MONFORT (1995), Simulation Based Econometric Methods.
  42. A. RUBINSTEIN (1996), Lectures on Modeling Bounded Rationality.
  43. J. RENEGAR (1999), A Mathematical View of Interior-Point Methods in Convex Optimization.
  44. B.D. BERNHEIM and M.D. WHINSTON (1999), Anticompetitive Exclusion and Foreclosure Through Vertical Agreements.
  45. D. BIENSTOCK (2001), Potential function methods for approximately solving linear programming problems: theory and practice.
  46. R. AMIR (2002), Supermodularity and complementarity in economics.
  47. R. WEISMANTEL (2006), Lectures on mixed nonlinear programming.

Related papers

Algorithms for Nonnegative Matrix Factorization
daniel lee

2000

Non-negative matrix factorization (NMF) has previously been shown to be a useful decomposition for multivariate data. Two different multiplicative algorithms for NMF are analyzed. They differ only slightly in the multiplicative factor used in the update rules. One algorithm can be shown to minimize the conventional least squares error while the other minimizes the generalized Kullback-Leibler divergence. The monotonic convergence of both algorithms can be proven using an auxiliary function analogous to that used for proving convergence of the Expectation-Maximization algorithm. The algorithms can also be interpreted as diagonally rescaled gradient descent, where the rescaling factor is optimally chosen to ensure convergence.

View PDFchevron_right
A Hybrid Algorithm for Low-rank Approximation of Nonnegative Matrix Factorization
Michael Antolovich

Neurocomputing, 2019

Nonnegative matrix factorization (NMF) is a recently developed method for data analysis. So far, most of known algorithms for NMF are based on alternating nonnegative least squares (ANLS) minimization of the squared Euclidean distance between the original data matrix and its low-rank approximation. In this paper, we first develop a new NMF algorithm, in which a Procrustes rotation and a nonnegative projection are alternately performed. The new algorithm converges very rapidly. Then, we propose a hybrid NMF (HNMF) algorithm that combines the new algorithm with the low-rank approximation based NMF (lraNMF) algorithm. Furthermore, we extend the HNMF algorithm to nonnegative Tucker decomposition (NTD), which leads to a hybrid NTD (HNTD) algorithm. The simulations verify that the HNMF algorithm performs well under various noise conditions, and HNTD has a comparable performance to the low-rank approximation based sequential NTD (lraSNTD) algorithm for sparse representation of tensor objects.

View PDFchevron_right
Accelerated Multiplicative Updates and Hierarchical ALS Algorithms for Nonnegative Matrix Factorization
Nicolas Gillis

Neural Computation, 2012

Nonnegative matrix factorization (NMF) is a data analysis technique used in a great variety of applications such as text mining, image processing, hyperspectral data analysis, computational biology, and clustering. In this paper, we consider two well-known algorithms designed to solve NMF problems, namely the multiplicative updates of Lee and Seung and the hierarchical alternating least squares of Cichocki et al. We propose a simple way to significantly accelerate their convergence, based on a careful analysis of the computational cost needed at each iteration. This acceleration technique can also be applied to other algorithms, which we illustrate on the projected gradient method of Lin. The efficiency of the accelerated algorithms is empirically demonstrated on image and text datasets, and compares favorably with a state-of-the-art alternating nonnegative least squares algorithm. Finally, we provide a theoretical argument based on the properties of NMF and its solutions that explains in particular the very good performance of HALS and its accelerated version observed in our numerical experiments.

View PDFchevron_right
Algorithms and applications for approximate nonnegative matrix factorization
memo herrera gonzalez

Computational Statistics Data Analysis, 2007

In this paper we discuss the development and use of low-rank approximate nonnegative matrix factorization (NMF) algorithms for feature extraction and identification in the fields of text mining and spectral data analysis. The evolution and convergence properties of hybrid methods based on both sparsity and smoothness constraints for the resulting nonnegative matrix factors are discussed. The interpretability of NMF outputs in specific contexts are provided along with opportunities for future work in the modification of NMF algorithms for large-scale and time-varying datasets.

View PDFchevron_right
A Unifying Framework and Comparison of Algorithms for Non‐negative Matrix Factorisation
Qianyun Guo

International Statistical Review

Non-negative matrix factorisation (NMF) is an increasingly popular unsupervised learning method. However, parameter estimation in the NMF model is a difficult high-dimensional optimisation problem. We consider algorithms of the alternating least squares type. Solutions to the least squares problem fall in two categories. The first category is iterative algorithms, which include algorithms such as the majorise-minimise (MM) algorithm, coordinate descent, gradient descent and the Févotte-Cemgil expectation-maximisation (FC-EM) algorithm. We introduce a new family of iterative updates based on a generalisation of the FC-EM algorithm. The coordinate descent, gradient descent and FC-EM algorithms are special cases of this new EM family of iterative procedures. Curiously, we show that the MM algorithm is never a member of our general EM algorithm. The second category is based on cone projection. We describe and prove a cone projection algorithm tailored to the non-negative least square problem. We compare the algorithms on a test case and on the problem of identifying mutational signatures in human cancer. We generally find that cone projection is an attractive choice. Furthermore, in the cancer application, we find that a mix-and-match strategy performs better than running each algorithm in isolation.

View PDFchevron_right

Related topics

  • Applied Mathematics
  • Image Processing
  • Alternating least squares
  • Applied Mathematics and Computat...
  • Numerical Analysis and Computati...
  • Nonnegative Matrix Factorization
  • Electrical and Electronic Engine...

    • Cited by

    Entire frame image display employing monotonic convergent nonnegative matrix factorization
    yogesh soniwal

    Journal of Real-Time Image Processing, 2017

    The average life spans of light-emitting diodes (LEDs) used in organic light-emitting diode displays are negatively influenced by high current operation in short duty cycles. Since the time average brightness of a pixel LED is a function only of the time integral of the delivered current, it is desirable to achieve brightness by adopting pixel addressing schemes that employ long duty cycles. Conventional addressing methods which cannot operate more than one row of pixels at a time inherently have very short duty cycles. We solve the problem by proposing entire frame addressing, made possible by a new image factorization called monotonic-nonnegative matrix factorization (M-NNMF). Using M-NNMF, we develop a converging image series representation of an arbitrary nonnegative matrix (image). Each element of the series has unit rank, which allows it to drive an entire frame simultaneously. Each application of the M-NNMF algorithm produces a dominant unit rank component for the series, and a residue image of higher rank, the input to the next iteration. We drive the display with these unit rank components, called sub-frames, each for a duration proportional to its energy. The sum of sub-frames approximates the original image and provides the same visual effect, due to inherent perceptual integration. M-NNMF, more efficient and less time complex compared to existing NNMF algorithms, is our primary contribution in this paper and is likely to find applications in many other situations. We also obtain an even faster converging series with a randomized version of M-NNMF. The proposed approach of entire frame image display is demonstrated on a wide variety of images.

    View PDFchevron_right
    580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved