Efficient Multiplicative updates for Support Vector Machines

2007

Abstract Nonnegative matrix factorization (NMF) is useful to find basis information of nonnegative data. Currently, multiplicative updates are a simple and popular way to find the factorization. However, for the common NMF approach of minimizing the Euclidean distance between approximate and true values, no proof has shown that multiplicative updates converge to a stationary point of the NMF optimization problem. Stationarity is important as it is a necessary condition of a local minimum.

Neural Computation, 2012

Nonnegative matrix factorization (NMF) is a data analysis technique used in a great variety of applications such as text mining, image processing, hyperspectral data analysis, computational biology, and clustering. In this paper, we consider two well-known algorithms designed to solve NMF problems, namely the multiplicative updates of Lee and Seung and the hierarchical alternating least squares of Cichocki et al. We propose a simple way to significantly accelerate their convergence, based on a careful analysis of the computational cost needed at each iteration. This acceleration technique can also be applied to other algorithms, which we illustrate on the projected gradient method of Lin. The efficiency of the accelerated algorithms is empirically demonstrated on image and text datasets, and compares favorably with a state-of-the-art alternating nonnegative least squares algorithm. Finally, we provide a theoretical argument based on the properties of NMF and its solutions that explains in particular the very good performance of HALS and its accelerated version observed in our numerical experiments.

2011

A multiplicative update for solving convex quadratic programming problems with nonnegativity constraints, which was proposed by Sha et al., has three advantages: 1) nonnegativity of solutions is automatically satisfied, 2) no parameter tuning is needed, and 3) implementation is easy because of simple update formula. However, the global convergence of the update is not always guaranteed. In this paper, we propose a modified version of the multiplicative update and prove its global convergence without any assumption on the problem. We also show experimentally that our modification affects neither the computation time nor the number of iterations

2010

Abstract Recent developments with indefinite SVM [11, 17, 5] have effectively demonstrated SVM classification with a non-positive kernel. However the question of efficiency still applies. In this paper, an efficient direct solver for SVM with non-positive kernel is proposed. The chosen approach is related to existing work on learning with kernel in Krein space. In this framework, it is shown that solving a learning problem is actually a problem of stabilization of the cost function instead of a minimization.

2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2011

Kernel nonnegative matrix factorization (KNMF) is a recent kernel extension of NMF, where matrix factorization is carried out in a reproducing kernel Hilbert space (RKHS) with a feature mapping φ(•). Given a data matrix X ∈ R m×n , KNMF seeks a decomposition, φ(X) ≈ U V ⊤ , where the basis matrix takes the form U = φ(X)W and parameters W ∈ R n×r + and V ∈ R n×r + are estimated without explicit knowledge of φ(•). As in most of kernel methods, the performance of KNMF also heavily depends on the choice of kernel. In order to alleviate the kernel selection problem when a single kernel is used, we present multiple kernel NMF (MKNMF) where two learning problems are jointly solved in unsupervised manner: (1) learning the best convex combination of kernel matrices; (2) learning parameters W and V. We formulate multiple kernel learning in MKNMF as a linear programming and estimate W and V using multiplicative updates as in KNMF. Experiments on benchmark face datasets confirm the high performance of MKNMF over several existing variants of NMF, in the task of feature extraction for face classification.

2000

Non-negative matrix factorization (NMF) has previously been shown to be a useful decomposition for multivariate data. Two different multiplicative algorithms for NMF are analyzed. They differ only slightly in the multiplicative factor used in the update rules. One algorithm can be shown to minimize the conventional least squares error while the other minimizes the generalized Kullback-Leibler divergence. The monotonic convergence of both algorithms can be proven using an auxiliary function analogous to that used for proving convergence of the Expectation-Maximization algorithm. The algorithms can also be interpreted as diagonally rescaled gradient descent, where the rescaling factor is optimally chosen to ensure convergence.

IEEE Transactions on Neural Networks, 2000

Multiplicative update algorithms have proved to be a great success in solving optimization problems with nonnegativity constraints, such as the famous nonnegative matrix factorization (NMF) and its many variants. However, despite several years of research on the topic, the understanding of their convergence properties is still to be improved. In this paper, we show that Lyapunov's stability theory provides a very enlightening viewpoint on the problem. We prove the exponential or asymptotic stability of the solutions to general optimization problems with nonnegative constraints, including the particular case of supervised NMF, and finally study the more difficult case of unsupervised NMF. The theoretical results presented in this paper are confirmed by numerical simulations involving both supervised and unsupervised NMF, and the convergence speed of NMF multiplicative updates is investigated.

2014 IEEE International Workshop on Machine Learning for Signal Processing (MLSP), 2014

The nonnegative matrix factorization (NMF) is widely used in signal and image processing, including bio-informatics, blind source separation and hyperspectral image analysis in remote sensing. A great challenge arises when dealing with nonlinear NMF. In this paper, we propose an efficient nonlinear NMF, which is based on kernel machines. As opposed to previous work, the proposed method does not suffer from the pre-image problem. We propose two iterative algorithms: an additive and a multiplicative update rule. Several extensions of the kernel-NMF are developed in order to take into account auxiliary structural constraints, such as smoothness, sparseness and spatial regularization. The relevance of the presented techniques is demonstrated in unmixing a synthetic hyperspectral image.

International Statistical Review

Non-negative matrix factorisation (NMF) is an increasingly popular unsupervised learning method. However, parameter estimation in the NMF model is a difficult high-dimensional optimisation problem. We consider algorithms of the alternating least squares type. Solutions to the least squares problem fall in two categories. The first category is iterative algorithms, which include algorithms such as the majorise-minimise (MM) algorithm, coordinate descent, gradient descent and the Févotte-Cemgil expectation-maximisation (FC-EM) algorithm. We introduce a new family of iterative updates based on a generalisation of the FC-EM algorithm. The coordinate descent, gradient descent and FC-EM algorithms are special cases of this new EM family of iterative procedures. Curiously, we show that the MM algorithm is never a member of our general EM algorithm. The second category is based on cone projection. We describe and prove a cone projection algorithm tailored to the non-negative least square problem. We compare the algorithms on a test case and on the problem of identifying mutational signatures in human cancer. We generally find that cone projection is an attractive choice. Furthermore, in the cancer application, we find that a mix-and-match strategy performs better than running each algorithm in isolation.

2011

Multiplicative update algorithms have encountered a great success to solve optimization problems with non-negativity constraints, such as the famous non-negative matrix factorization (NMF) and its many variants. However, despite several years of research on the topic, the of their convergence properties is still to be improved. In this paper, we show that Lyapunov's stability theory provides a very enlightening viewpoint on the problem. We prove the stability of supervised NMF and study the more difficult case of unsupervised NMF. Numerical simulations illustrate those theoretical results, and the convergence speed of NMF multiplicative updates is analyzed.