An efficient method for simplifying support vector machines

2001

Abstract This paper describes a simple modification to the sequential minimal optimisation (SMO) training algorithm for support vector machine (SVM) classifiers, reducing training time at the expense of a small increase in memory used proportional to the number of training patterns. Results obtained on real-world pattern recognition tasks indicate that the proposed modification can more than halve the average training time.

2006

Abstract There are well-established methods for reducing the number of support vectors in a trained binary support vector machine, often with minimal impact on accuracy. We show how reduced-set methods can be applied to multiclass SVMs made up of several binary SVMs, with significantly better results than reducing each binary SVM independently.

Journal of Machine Learning Research, 2006

Support vector machines (SVMs), though accurate, are not preferred in applications requiring great classification speed, due to the number of support vectors being large. To overcome this problem we devise a primal method with the following properties: (1) it decouples the idea of basis functions from the concept of support vectors; (2) it greedily finds a set of kernel basis functions of a specified maximum size (d max ) to approximate the SVM primal cost function well; (3) it is efficient and roughly scales as O(nd 2 max ) where n is the number of training examples; and, (4) the number of basis functions it requires to achieve an accuracy close to the SVM accuracy is usually far less than the number of SVM support vectors.

2004

Support Vector Machines (SVMs) have gained wide acceptance because of the high generalization ability for a wide range of classification applications. Although SVMs have shown potential and promising performance in classification, they have been limited by speed particularly when the training data set is large. The hyper plane constructed by SVM is dependent on only a portion of the training samples called support vectors that lie close to the decision boundary (hyper plane). Thus, removing any training samples that are not relevant to support vectors might have no effect on building the proper decision function. We propose the use of clustering techniques such as K-mean to find initial clusters that are further altered to identify non-relevant samples in deciding the decision boundary for SVM. This will help to reduce the number of training samples for SVM without degrading the classification result.

Proceedings of the 23rd international conference on Machine learning - ICML '06, 2006

In machine learning problems with tens of thousands of features and only dozens or hundreds of independent training examples, dimensionality reduction is essential for good learning performance. In previous work, many researchers have treated the learning problem in two separate phases: first use an algorithm such as singular value decomposition to reduce the dimensionality of the data set, and then use a classification algorithm such as naïve Bayes or support vector machines to learn a classifier. We demonstrate that it is possible to combine the two goals of dimensionality reduction and classification into a single learning objective, and present a novel and efficient algorithm which optimizes this objective directly. We present experimental results in fMRI analysis which show that we can achieve better learning performance and lower-dimensional representations than two-phase approaches can.

2015

We develop an approach for feature elimination in support vector machines (and empirical risk minimization), based on recursive elimination of features. We present theoretical properties of this method and show that this is uniformly consistent in finding the correct feature space under certain generalized assumptions. We present case studies to show that the assumptions are met in most practical situations and also present simulation studies to demonstrate performance of the proposed approach. FEATURE ELIMINATION IN SUPPORT VECTOR MACHINES AND EMPIRICAL RISK MINIMIZATION By Sayan Dasgupta1, Yair Goldberg2 and Michael R. Kosorok1 1The University of North Carolina at Chapel Hill 2University of Haifa We develop an approach for feature elimination in support vector machines (and empirical risk minimization), based on recursive elimination of features. We present theoretical properties of this method and show that this is uniformly consistent in finding the correct feature space under c...

2008 IEEE International Conference on Systems, Man and Cybernetics, 2008

Maximizing the classification performance of the training data is a typical procedure in training a classifier. It is well known that training a Support Vector Machine (SVM) requires the solution of an enormous quadratic programming (QP) optimization problem. Serious challenges appeared in the training dilemma due to immense training and this could be solved using Sequential Minimal Optimization (SMO). This paper investigates the performance of SMO solver in term of CPU time, number of support vector and decision boundaries when applied in a 2-dimensional datasets. Next, the chunking algorithm is employed for comparison purpose. Initial results demonstrated that the SMO algorithm could enhance the performance of the training dataset. Both algorithms illustrated similar patterns from the decision boundaries attained. Classification rate achieved by both solvers are superb.

Object recognition supported by user interaction for service robots, 2002

has been proved to be a powerful tool for solving practical pattern recognition problems based on learning from data. Due to large number of support vectors learnt from huge amount of training data the SVM becomes too computational intensive to many critical problems. In this paper we develop a reliable reduced set vectors method to speed up the SVM with Gaussian kernel. A set of reduced vector pairs (RVPs) are calculated from the support vectors. In the case of face detection, by considering the RVPs sequentially, if at any point a window is deemed too unlikely to cease the sequential evaluation, obviating the need to evaluate the remaining RVPs so that we only need to apply a subset of the RVPs to eliminate things that are obviously not a face.

18th International Conference on Pattern Recognition (ICPR'06), 2006

In many classification applications, Support Vector Machines (SVMs) have proven to be highly performing and easy to handle classifiers with very good generalization abilities. However, one drawback of the SVM is its rather high classification complexity which scales linearly with the number of Support Vectors (SVs). This is due to the fact that for the classification of one sample, the kernel function has to be evaluated for all SVs. To speed up classification, different approaches have been published, most which of try to reduce the number of SVs. In our work, which is especially suitable for very large datasets, we follow a different approach: as we showed in [12], it is effectively possible to approximate large SVM problems by decomposing the original problem into linear subproblems, where each subproblem can be evaluated in Ω(1). This approach is especially successful, when the assumption holds that a large classification problem can be split into mainly easy and only a few hard subproblems. On standard benchmark datasets, this approach achieved great speedups while suffering only sightly in terms of classification accuracy and generalization ability. In this contribution, we extend the methods introduced in [12] using not only linear, but also non-linear subproblems for the decomposition of the original problem which further increases the classification performance with only a little loss in terms of speed. An implementation of our method is available in [13]. Due to page limitations, we had to move some of theoretic details (e.g. proofs) and extensive experimental results to a technical report [14].

… : Formal Models and …, 2005

This paper presents a comparison of different initialization algorithms joint with decomposition methods, in order to reduce the training time of Support Vector Machines (SVMs). Training a SVM involves the solution of a quadratic optimization problem (QP).The QP problem is very resource consuming (computational time and computational memory), because the quadratic form is dense and the memory requirements grow square the number of data points. The SVM-QP problem can be solved by several optimization strategies but, for large scale applications, they must be combined with decomposition algorithms that breaks up the entire SVM-QP problem into a series of smaller ones. The support vectors found in the training of SVMs represent a small subgroup of the training patterns. Some algorithms are used to initilizate the SVMs, making a fast approximation of the points standing for support vectors, to train the SVM only with those data. Combination of these initializations algorithms and the decomposition approach, coupled with an QP solver specially arranged for the SVM-QP problem, are compared using some well-known benchmarks in order to show their capabilities.