Online Support Vector Regression

Francesco Parrella

Outline

Online Support Vector Regression

Francesco Parrella

2007

Abstract

Many approaches for obtaining systems with intelligent behavior are based on components that learn automatically from previous experience. The development of these learning techniques is the objective of the area of research known as machine learning. During the last decade, researchers have produced numerous and outstanding advances in this area, boosted by the successful application of machine learning techniques. This thesis presents one of this techniques, an online version of the algorithm for training the support vector machine for regression and also how it has been extended in order to be more flexible for the hyper parameter estimation. Furthermore the algorithm has been compared with a batch implementation

Figures (86)

learning. First of all, we will start with the biological definition: Learning is the process of acquiring knowledge, skills, attitudes, or values, through Nowadays, the meaning of ” learning” has many definitions. The purpose of this first

knowledge is the lowest order of cognition and evaluation the highest: e Knowledge is the memory of previously-learnt materials such as facts, terms,

a sequence of 0 and 1, without semantic understanding. Figure 2.3: How machines see characters

analysis, voice recognition, and many others. Figure 2.4: Machine learning applications: virus detection and speech recognition

Figure 2.5: Supervised learning algorithms: a decision tree, a neural network and a

Figure 2.6: Unsupervised algorithm example: clustering 2.4. The machine learning process 4.3 Semi-Supervised learning

using a different, and ideally independent, set of features for each example. Figure 2.7: Semi-supervised learning example

ment learning. Figure 2.8: Reinforced learning example The principal algorithms are temporal difference learning and actor critic reinforce-

and output makes up how they should be catalogued. Each example can be written as a pair (input, output) where input is the data s« Mathematically, examples can be regarded as a pair (x, y) where x is a vector of real numbers and output can be a boolean value (like as yes/no, 1/-1, true/false),

Figure 3.2: From a conceptual to a mathematical view yenerates the function which interpolates, to the best of its capability these series or a real number. In the first case, we speak of a problem of classification, in the

suppose we know the value of these figurative companies: Zoogle, Ciat, Carmalat

Figure 3.4: The yellow line shows the result of the training all the information we need to make the SVM work. The results obtained are as whereas Carmalat and Karmacotto have not had much success. We therefore have

we just have a look at its position in the chart to get a good idea: Because of the fact that it is positioned above the yellow line, the SVM considers

a little higher or lower?”. In theory, all these are possible solutions: The observer may ask himself ” why is the line exactly there where it is,why isn’t it 3 A more detailed analysis

solution and examples is called margin. In our case, it can be represented as: Suppose that the possible outputs are 1 and -1, they are shown on those two

Figure 3.8: Coyota is not a good stock, but is placed near some good stocks. Now happen. For example, let us suppose that added to our list of 4 stocks is the Coyota In the former example, the data were linearly separable or, in other words, it exists If Coyota isn’t a good stock we would not be able to find a line to divide it.

case of the Coyota stock, we can visualize the probable solution like this: Figure 3.9: A solution at the Coyota problem other problems require more elaborate and complex solutions. For example, in the But how can one obtain different functions from a line? How can we introduce

it becomes much more complex, a not linear function. Figure 3.11: From features space back to linear space

Figure 3.10: From linear to features space 3.5. More generic Support Vector Machines

Figure 3.12: A difficult problem solved using a gaussian kernel 3.5. More generic Support Vector Machines

also make wrong classifications, like provided by this example: Figure 3.13: An example of a bad parameters setting

3.6. SVR - Support Vector Machines for Regression It is obviously not possible to know when its the best moment to buy or sell

many websites that have as a function to analyze the financial market. Here’s a1 in the training (dates, quotations of the stock on those date) can be extracted from

part of the error is tolerated. individualizing the hyperplane which maximizes the margin, keeping in mind that The figure shows how the error of SVR is calculated. Up until the threshold e,

Coming back to the mathematical point of view, the quadratic optimization

up of pairs (day, market quotes) which correspond to this plot: To make the SVR work, let’s make the tolerance margin at 0.01 over the fina

over information that give us the best solution. The result of this would be: The yellow line corresponds to prevision of the SVR in the following days. In

Figure 3.20: When buy and sell the stocks

and based upon certain predictions, when to sell and buy the stock. Figure 3.21: A virtual stock broker build using SVM and SVR knowledge to decide whether a particular stock will be a good or bad investment.

The constant C determines the trade-off between the flatness of the function and the

all the constraints, introducing some Lagrange multipliers:

Finally also the C parameter can be deleted, using the derivative constraints And deleting some unnecessary parts:

As before, we can compute another Lagrange function that includes all the con- straints, introducing other Lagrange multipliers: After many math passages, minimizing a, a* values, there is another dual optimiza-

Like exposed above, this is an optimization problem with a convex domain. The

All we need to find is the relation of h(x) and ¢ at the changing of a:

Replacing the variations of 6 and b with the results obtained in the equation (4.13): In this way, AO; and Ab values can be updated by computing 3, and A(x;) value: can be updated by computing 7.

h(x) as follows: In this image, each circle corresponds to a sample. The red circles are error sam- ples, the yellow-ones are support samples and the green-ones are remaining samples.

The first possibility is that the new sample error is less than €. This is the easiest

Otherwise, when the error is bigger than € the situation is more complicated:

In this situation 0. or h(x.) value of the new sample changes until it reaches the

The problem is that these variations influence other samples: At the end of the updating procedure, if variation is too big, some samples could Notice that the arrow represents the sample movements of 6; or h(x;) components.

4.9. From Support to Error Set The direction of a sample is determined by the sign of 3; for support samples and

When a sample goes from support to error set, his 0; from 0 < |0;| < C becomes

10 From Support to Remaining Set When a sample goes from support to remaining set, his 0; from 0 < |6;| < C becomes For support samples, the updating formula is:

When a sample goes from error to support set, his |h(x;)| from > € becomes = e. For error samples, the updating formula is:

From Remaining to Support Set When a sample goes from remaining to support set, his |h(x;)| from < € becomes For error samples, the updating formula is:

Before a new sample is added, all the samples that requires a less variation are

In the next paragraphs are listed all the possible variations. Note that if KKT conditions aren’t violated, this part of the algorithm isn’t used. them with new possible moves permitted only when a sample exit from its set.

An error sample becomes invalid when its |h(x;)| becomes < e. In this case the sample, is neither an error sample, neither a remaining sample,

4.14.3 Invalid Remaining Samples In this case the sample, is neither a remaining sample, neither an error sample,

Variation value is: The second variation measures the distance of new sample from error set. new sample reaches |e] and consequently the SupportSet. Variation value is:

The third variation measures the distance of each Support sample to Error or Re-

Variation values are: The fourth variation measures the distance of each error sample to SupportSet. position inside the SupportSet.

The last variation measures the distance of each remaining sample to SupportSet.

one should recompute also @ and 4; as follows: The remove procedure delete the line and the column of the erased sample anc Otherwise, if the new sample is moved from Error or RemainingSet to SupportSet, update the others:

Figure 7.1: Samples trained compared with iteration count

So, we can compare the exact time to add a new sample using the two methods: Figure 7.3: OnlineSVR and LibSVM compared in an online training 10t the total speed of the training.

Table 7.1: Training parameters of the stabilization test The table shows the parameters of the SVR used during the stabilization test. all the parameters and compared the results.

be changed to avoid overfitting and underfitting. Figure 7.4: Stabilization and new learning time when C is increased

Figure 7.5: Stabilization and new learning time when C is decreased

we were to restart the training. Figure 7.6: Stabilization and new learning time when € is increased

Figure 7.7: Stabilization and new learning time when « is decreased

Figure 7.8: Stabilization and new learning time when kernel param is increased

Figure 7.9: Stabilization and new learning time when kernel param is decreased

up by a head and an arm: Figure 7.10: The James robot

Here are a summary scheme: Figure 7.11: The training scheme

Figure 7.12: The first graph shows the error of a sample before learning, the second one the mean error and the third one the standard deviation To measure the training worth we have observed the mean and standard deviation

Here are the result obtained from the first motor: Figure 7.13: Motor Arm 1 results

The second motor: Figure 7.14: Motor Arm 2 results

Figure 7.16: The results are not as good as we would

Figure 7.17: Noisy problem: these two arm positions for the robot are the same. were somewhat suspicious. For example two positions of the robot seemed identical:

the training and test sets. It is very useful for manual estimation of the parameters. It also plots the errors of

available on http://onlinesvr.altervista.org.

References (30)

TrainingSet {x i , y i , i = 1..l}
Wheights θ i , i = 1..l
TrainingSet partition into SuppotSet(S) , ErrorSet(E) and RemainingSet(R)
Params: , C, KernelType and KernelParam
R Matrix
5.1 Update the values β and γ 5.2 Find Least Variations (L c , L s , L e , L r )
3 Find Min Variation ∆θ c = min(L c , L s , L e , L r )
4 Let Flag the case number that determinates ∆θ c (L c = 1, L s = 2, L e = 3, L r = 4)
6 Update θ c , θ i , i = 1..l and b 5.7 Update h(x i ), i ∈ E ∪ R
R Matrix At the beginning, Online SVR check the set of the sample. If it's in the Remain- ingSet, it is easy to remove. Otherwise it should be removed from its sample and start a cycle that end only A.5 OnlineSVR Web Site It contains all the documentation of the algorithm and the code produced. It is available on http://onlinesvr.altervista.org. Cherkassky and Ma (2002b)
"In the support-vector networks algorithm one can control the trade-off between complexity of decision rule and frequency of error by changing the parameter C,..." Cortes and Vapnik "There are a number of learning parameters that can be utilized in constructing SV machines for regression. The two most relevant are the insensitivity zone e and the penalty parameter C, which determines the trade-off between the training error and VC dimension of the model. Both parameters are chosen by the user." Kecman (2001) The parameter C controls the trade off between errors of the SVM on training data and margin maximization (C = [infinity] leads to hard margin SVM). Rychetsky (2001)
"The parameter C controls the trade-off between the margin and the size of the slack variables." Shawe-Taylor and Cristianini (2004)
"[Tuning the parameter C] In practice the parameter C is varied through a wide range of values and the optimal performance assessed using a separate validation set or a technique known as cross-validation for verifying performance using only a training set." Shawe-Taylor and Cristianini (2004) "...the parameter C has no intuitive meaning." Shawe-Taylor and Cristianini (2004)
"The factor C in is a parameter that allows one to trade off training error vs. model complexity. A small value for C will increase the number of training errors, Bibliography C.1 Incremental Support Vector Machines
Accurate Online Support Vector Regression, Ma, Theiler, Perkins, Article, 2003 T his is the article where I started my research.
Incremental and Decremental Support Vector Machine Learning, Cauwenbergh, Poggio, Article, 2001 S tart point of all incremental support vector algorithms.
Incremental Support Vector Learning: Analysis, Implementation and Appli- cations, Laskov, Gehl, Krauger, Muller, Article, 2001
Article rich of demostrations about incremental classification. It includes also some implementation tricks for an efficient implementation.
Online SVM Learning: from Classification to Data Description and Back, Tax, Laskov, Article, 2003 U seful to understand how the algorithm converge.
C.2 Support Vector Machines
A tutorial on support vector regression, Smola, Schoelkopf, Article, 1998 S tart point for support vector regression.
Support Vector Machines, Taylor and Cristianini, Book, 2000 T heory and concepts abot support vector machines.
Support Vector Machines for Classification and Regression, Gunn, Article, 1998
C omplete article about support vector machines: contains theory, examples and the matlab code of the implemenetation.
LubSVM -A Library for Support Vector Machines, Chang and Lin, Web Site, 2006 http://www.csie.ntu.edu.tw/ cjlin/libsvm/
Support Vector Machines, Grudic, Article, 2004 I ntroduction to support vector machines.
Support Vector Machines, Maniezzo, Article, 2004 I ntroduction to support vector machines.
UCI Machine Learning Repository, Various authors, Web Site, 2004 http://www.ics.uci.edu/ mlearn/MLRepository.html
Wikipedia, Various authors, Online Repository, 1998 http://en.wikipedia.org/wiki/Learning
SVM Parameters, Various authors, Web Site, 2004 http://www.svms.org/parameters/

Online Support Vector Regression

Sign up for access to the world's latest research

Abstract

Related papers

References (30)

Related papers

Related topics