An Improved Way to Make Large-Scale SVR Learning Practical

We ﬁrst put forward a new algorithm of reduced support vector regression (RSVR) and adopt a new approach to make a similar mathematical form as that of support vector classiﬁcation. Then we describe a fast training algorithm for simpliﬁed support vector regression, sequential minimal optimization (SMO) which was used to train SVM before. Experiments prove that this new method converges considerably faster than other methods that require the presence of a substantial amount of the data in memory


INTRODUCTION
In the last few years, there has been a surge of interest in support vector machine (SVM) [1].SVM has empirically been shown to give good generalization performance on a wide variety of problems.However, the use of SVM is still limited to a small group of researchers.One possible reason is that training algorithms for SVM are slow, especially for large problems.Another explanation is that SVM training algorithms are complex, subtle, and sometimes difficult to implement.
In 1997, a theorem [2] was proved that introduced a whole new family of SVM training procedures.In a nutshell, Osuna's theorem showed that the global SVM training problem can be broken down into a sequence of smaller subproblems and that optimizing each subproblem minimizes the original quadratic problem (QP).Even more recently, the sequential minimal optimization (SMO) algorithm was introduced [3,4] as an extreme example of Osuna's theorem in practice.Because SMO uses a subproblem of size two, each subproblem has an analytical solution.Thus, for the first time, SVM could be optimized without a QP solver.
In addition to SMO, other new methods [5,6] have been proposed for optimizing SVM online without a QP solver.While these other online methods hold great promise, SMO is the only online SVM optimizer that explicitly exploits the quadratic form of the objective function and simultaneously uses the analytical solution of the size two cases.
Support vector regression (SVR) have nearly the same situation as SVM.In 1998, Smola and Schölkopf [7] gave an overview of the basic idea underlying SVMs for regression and function estimation.They also generalized SMO so that it can handle regression problems.A detailed discussion can also be found in Keerthi [8] and Flake [9].Because one has to consider four variables, α i , α * i , α j , and α * j , in the regression, the training algorithm actually becomes very complex, especially, when data is nonsparse and when there are many support vectors in the solution-as is often the case in regression-because kernel function evaluations tend to dominate the runtime in this case, most of these variables do not converge to zero and its rate of convergence slows down dramatically.
In this work, we propose a new way to make SVR-a new regression technique based on the structural risk minimization principle-has a similar mathematical form as that of support vector classification, and derives a generalization of SMO to handle regression problems.Simulation results indicate that the modification to SMO for regression problem yields dramatic runtime improvements.
We now briefly outline the contents of the paper.In Section 2, we describe previous works for train SVM and SVR.In Section 3, we outline our reduced SVR approach and simplify its mathematical form so that we can express SVM and SVR in a same form.Then we describe a fast training algorithm for simplified SVR, sequential minimal optimization.Section 4 gives computational and graphical results that show the effectiveness and power of Reduced Support Vector Recognition (RSVR).Section 5 concludes the paper.

SMO for SVM
The QP problem to train an SVM is shown below: The QP problem in (1) is solved by the SMO algorithm.A point is an optimal point of (1) if and only if the Karush-Kuhn-Tucker (KKT) conditions are fulfilled and Q i j = y i y j k( x i , x j ) is positive semidefinite.Such a point may be a nonunique and nonisolated optimum.The KKT conditions are particularly simple; the QP problem is solved when, for all i, (2) Unlike other methods, SMO chooses to solve the smallest possible optimization problem at every step.In each time, SMO chooses two Lagrange multipliers to jointly optimize, finds the optimal values for these multipliers, and updates the SVM to reflect the new optimal values.The advantage of SMO lies in the fact that solving for two Lagrange multipliers can be done analytically.Thus, an entire inner iteration due to numerical QP optimization is avoided.
In addition, SMO does not require extra matrix storage.Thus, very large SVM training problems can fit inside of the memory of an ordinary personal computer or workstation.
Because of these advantages, SMO is well suited for training SVM and becomes the most popular training algorithm.

Training algorithms for SVR
Chunking, which was introduced in [10], relies on the observation that only the SVs are relevant for the final form of the hypothesis.Therefore, the large QP problem can be broken down into a series of smaller QP problems, whose ultimate goal is to identify all of the nonzero Lagrange multipliers and discard all of the zero Lagrange multipliers.Chunking seriously reduces the size of the matrix from the number of training examples squared to approximately the number of nonzero Lagrange multipliers squared.However, chunking still may not handle large-scale training problems, since even this reduced matrix may not fit into memory.
Osuna [2,11] suggested a new strategy for solving the QP problem and showed that the large QP problem can be broken down into a series of smaller QP subproblems.As long as at least one example that violates the KKT conditions is added to the examples for the previous subproblem, each step reduces the overall objective function and maintains a feasible point that obeys all of the constraints.Therefore, a sequence of QP subproblems that always add at least one violator will asymptotically converge.
Based on the SMO, Smola [7] generalized SMO to train SVR.Consider the constrained optimization problem for two indices, say (i, j).Pattern dependent regularization means that C i may be different for every pattern (possibly even different for α i , α * i ).For regression, one has to consider four different cases, (α i , α j ), (α i , α * j ), (α * i , α j ), and (α * i , α * j ).Thus, one obtains from the summation constraint where L, H are defined as the boundary of feasible regions for regression.SMO has better scaling with training set size than chunking for all data sets and kernels tried.Also, the memory footprint of SMO grows only linearly with the training set size.SMO should thus perform well on the largest problems, because it scales very well.

REDUCED SVR AND ITS SMO ALGORITHM
Most of those already existing training methods are originally designed to only be applicable to SVM.Compared with SVM, SVR has more complicated form.For SVR, there are two sets of slack variables, (ξ 1 , . . ., ξ n ) and (ξ * 1 , . . ., ξ * n ), and their corresponding dual variables, (α 1 , . . ., α n ) and (α * 1 , . . ., α * n ).The analytical solution to the size-two QP problems must be generalized in order to work on regression problems.Even though Smola has generalized SMO to handle regression problems, one has to distinguish four different cases, (α i , α j ), (α i , α * j ), (α * i , α j ), and (α * i α * j ).This makes the training algorithm more complicated and difficult to implement.In this paper, we propose a new way to make SVR have the similar mathematical form as that of support vector classification, and derive a generalization of SMO to handle regression problems.

RSVR and its simplified formulation
Recently, the RSVM [12] was proposed as an alternate of the standard SVM.Similar to (1), we now use a different regression objective which not only suppresses the parameter w, but also suppresses b in our nonlinear formulation.Here we first introduce an additional term b 2 /2 to SVR and outline the key modifications from standard SVR to RSVR.Hence we arrive at the formulation stated as follows, ( It is interesting to note that very frequently the standard SVR problem and our variant (3) give the same w.In fact, from [12] we can see the result which gives sufficient conditions that ensure that every solution of RSVM is also a solution of standard SVM for a possibly larger C. The same conclusion can be generalized to the RSVR case easily.Later we will show computationally that this reformulation of the conventional SVM formulation yields similar results to SVR.
By introducing two dual sets of variables, we construct a Lagrange function from both the objective function and the corresponding constraints.It can be shown that this function has a saddle point with respect to the primal and dual variables at the optimal solution It is understood that the dual variables in (4) have to satisfy positivity constraints, that is, α i , α * i , η i , η * i ≥ 0. It follows from the saddle point condition that the partial derivatives of L with respect to the primal variables (w, b, ξ i , ξ * i ) have to vanish for optimality.
(5) Substituting ( 5) into (4) yields the dual optimization problem minimize 1 2 The main reason for introducing our variant (3) of the RSVR is that its dual (6) does not contain an equality constraint, as does the dual optimization problem of original SVR.This enables us to apply in a straightforward manner the effective matrix splitting methods, such as those of [13], that process one constraint of (3) at a time through its dual variable, without the complication of having to enforce an equality constraint at each step on the dual variable α.This permits us to process massive data without bringing it all into fast memory. Define Thus, ( 6) can be expressed in a way, If we ignore the difference of matrix dimension, ( 8) and (2) will have the similar mathematical form.So, many training algorithms that were used in SVM can be used in RSVR.Thus, we obtain an expression which can be evaluated in terms of dot products between the pattern to be regressed and the support vectors To compute the threshold b, we take into account that due to (5), the threshold can for instance be obtained by (10)

Analytic solution for RSVR
Note that there is little difference between generalized RSVR and SVR.The dual (6) does not contain an equality constraint, but we can take advantage of (10) to solve this problem.Here, b is regarded as a constant, while for conventional SVR b equals to zero.Each step of SMO will optimize two Lagrange multipliers.Without loss of generality, let these two multipliers be α 1 and α 2 .The objective function from (8) can thus be written as where and the variables with "old" superscripts indicate values at the end of the previous iteration.W constant are terms that do not depend on either α 1 or α 2 .Each step will find the maximum along the line defined by the linear equality in (10).That linear equality constraint can be expressed as The objective function can be expressed in terms of α 2 alone, The stationary point of the objective function is at If the second derivate along the linear equality constraint is positive, then the maximum of the objective function can be expressed as Expanding the equations for r, u, and v yields Then Then the following bounds apply to α 2 : .
By solving (8) for Lagrange multipliers α, b can be computed as (10).After each step, b is recomputed, so that the KKT conditions are fulfilled for the optimization problem.

EXPERIMENTAL RESULTS
The RSVR algorithm is tested against the standard SVR training with chunking algorithm and against Smola's SMO method on a series of benchmarks.The RSVR, SMO, and chunking are all written in C++, using Microsoft's Visual C++ 6.0 compiler.Joachims' package SVM light1  The chunking algorithm uses the projected conjugate gradient algorithm as its QP solver, as suggested by Burges [1].All algorithms use sparse dot product code and kernel caching.Both SMO and chunking share folded linear SVM code.
In Figure 1b, we can also observe the action of Lagrange multipliers acting as forces (α i , α * i ) pulling and pushing the regression inside the ε-tube.These forces, however, can only be applied to the samples where the regression touches or even exceeds the predetermined tube.This directly accords with the illustration of the KKT-conditions, either the regression lies inside the tube (hence the conditions are satisfied with a margin), and Lagrange multipliers are 0, or the condition is exactly met and forces have to be applied to α i = 0 or α * i = 0 to keep the constraints satisfied.This observation proves that the RSVR method can handle regression problems successfully.
In Table 1, we can see that the SVM trained with other various methods have nearly the same approximation accuracy.However, in this experiment, we can see that the testing accuracy of RSVR is little lower than traditional SVR.
Moreover, as the training efficiency is the main motivation of RSVR, we would like to discuss its different implementations and compare their training time with regular SVR.
Experiment 2. In order to compare the time consume of different training methods on massive data sets, we test these algorithms on three real-world data sets.
In this experiment, we adopt the same data sets used in [14].In this experiment, we use the same kernel with C = 3000 and kernel parameters are shown in Table 2.Here we compare the programs on three different tasks that are stated as follows.

Kin
This data set represents a realistic simulation of the forward dynamics of an 8 link all-revolute robot arm.The task is to predict the distance of the end-effecter from a target, given  features like joint positions, twist angles, etc.The first data is of size 650.

Sunspots
Using a series representing the number of sunspots per day, we created one input/output pair for each day, the yearly average of the year starting the next day had to be predicted using the 12 previous yearly averages.This data set is of size 4000.

Forest
This data set is a classification task with 7 classes [14], where the first 20000 examples are used here.We transformed it into a regression task where the goal is to predict +1 for examples of class 2 and −1 for the other examples.Experiment 3. In this experiment, we will use the RSVR trained by SMO to predict time series data set.Here we adopt Greenwich's sunspot data.The kernel parameters are C = 3000, δ = 500, and ε = 10.We can also gain these data from Greenwich's homepage (http://science.msfc.nasa.gov/ssl/pad/solar/greenwch.htm).We use historic sunspot data to predict future sunspot data.Figure 2 shows the comparison between real sunspot data and predicted sunspot data.This illustrates that the SVM give good prediction to sunspot.This experiment proves that the RSVR trained by SMO algorithm can be used in practical problems successfully.

CONCLUSION
We have discussed the implementations of RSVR and its SMO fast training algorithm.Compared with Smola's SMO algorithm, we successfully reduce the variables from four to two.This reduces the complexity of training algorithm greatly and makes it easy to implement.Also we compare it with conventional SVR.Experiments indicate that in general the test accuracy of RSVR is little worse than that of the standard SVR.For the training time which is the main motivation of RSVR, we show that, based on the current implementation techniques, RSVR will be faster than regular SVR on large data set problems or some difficult cases with many support vectors.Therefore, for medium-size problems, standard SVR should be used, but for large problems, RSVR can effectively restrict the number of support vectors and can be an appealing alternate.Thus, for very large problems it is appropriate to try the first.

Figure 2 :
Figure 2: Comparison between real sunspot data (solid line) and predicted sunspot data (dashed line).

Table 1 :
Approximation effect of SVR using various methods.

Table 2 :
Comparison on various data sets.

Table 2
illustrates the time consume, the training set size, and the number of support vectors for different training algorithms.In each data set, the objective values of training error are the same.Here we can see that with data set increase, the difference of training time among these training algorithms also increases greatly.When the size of data set reaches 20000, the training time needed by Chunking and SVM light is more than 11 times than that of RSVR.Here we define the training error to be the MSE over the training data set.