Kernel Machines
Non-parallel support vector classifiers with different loss functions [Neurocomputing 2014][PDF][Github]
This paper introduces a general framework of non-parallel support vector machines, which involves a regularization term, a scatter loss and a misclassification loss. When dealing with binary problems, the framework with proper losses covers some existing non-parallel classifiers, such as multisurface proximal support vector machine via generalized eigenvalues, twin support vector machines, and its least squares version. The possibility of incorporating different existing scatter and misclassification loss functions into the general framework is discussed. Moreover, in contrast with the mentioned methods, which applies kernel- generated surface, we directly apply the kernel trick in the dual and then obtain nonparametric models. Therefore, one does not need to formulate two different primal problems for the linear and nonlinear kernel respectively. In addition, experimental results are given to illustrate the performance of different loss functions.
In many contexts, learning from few labeled and large amount of unlabeled data points is highly desirable. Here MSS-KSC approach, an which follows an optimization problem formulated in the primal-dual settings where the available labeled data points are incorporated into the core model via a regularization term is proposed. The core model is Kernel Spectral Clustering (KSC), a completely unsupervised algorithm. MSS-KSC is a kernel based model with out-of-sample extension property. It addresses both multi-class semi-supervised classification and semi-supervised clustering. It realizes low embedding dimension to reveal the existing number of clusters. Thanks to the proper model selection scheme, it can detect hidden micro-clusters. The method has been successfully applied in several areas including: classification, clustering, image segmentation, community detention.
The use of indefinite kernels has attracted many research interests in recent years due to their flexibility. They do not possess the usual restrictions of being positive definite as in the traditional study of kernel methods. This paper introduces the indefinite unsupervised and semi-supervised learning in the framework of least squares support vector machines (LS-SVM). The analysis is provided for both unsupervised and semi-supervised models, i.e., Kernel Spectral Clustering (KSC) and Multi-Class Semi-Supervised Kernel Spectral Clustering (MSS-KSC). In indefinite KSC models one solves an eigenvalue problem whereas indefinite MSS-KSC finds the solution by solving a linear system of equations. For the proposed indefinite models, we give the feature space interpretation, which is theoretically important, especially for the scalability using Nyström approximation. Experimental results on several real-life datasets are given to illustrate the efficiency of the proposed indefinite kernel spectral learning.
As the simplest extension of linear classifiers, piecewise linear (PWL) classifiers have attracted a lot of attention, because of their simplicity and classification capability. In this paper, a PWL feature mapping is introduced by investigating the property of the PWL classification boundary. Then support vector machines (SVM) with PWL feature mappings are proposed, named as PWL-SVMs. In this paper, it is shown that some widely used classifiers, such as k-nearest neighbor, adaptive boosting of linear classifier and intersection kernel support vector machine, can be represented by the proposed feature mapping. That means the proposed PWL-SVMs at least can archive the performance of other PWL classifiers. Moreover, PWL-SVMs enjoy good properties of SVM and the performance on numerical experiments illustrates the effectiveness. Then some extensions are discussed and the application of PWL-SVMs can be expected.
Robust support vector machines for classification with nonconvex and smooth losses[Nueral Computation 2016][PDF]
This letter addresses the robustness problem when learning a large margin classifier in the presence of label noise. In our study, we achieve this purpose by proposing robustified large margin support vector machines. The robustness of the proposed robust support vector classifiers (RSVC), which is interpreted from a weighted viewpoint in this work, is due to the use of nonconvex classification losses. Besides the robustness, we also show that the proposed RSCV is simultaneously smooth, which again benefits from using smooth classification losses. The idea of proposing RSVC comes from M-estimation in statistics since the proposed robust and smooth classification losses can be taken as one-sided cost functions in robust statistics. Its Fisher consistency property and generalization ability are also investigated. Besides the robustness and smoothness, another nice property of RSVC lies in the fact that its solution can be obtained by solving weighted squared hinge loss–based support vector machine problems iteratively. We further show that in each iteration, it is a quadratic programming problem in its dual space and can be solved by using state-of-the-art methods. We thus propose an iteratively reweighted type algorithm and provide a constructive proof of its convergence to a stationary point. Effectiveness of the proposed classifiers is verified on both artificial and real data sets.
Small data materials design with machine learning: When the average model knows best[Journal of Applied Physics, 2020][PDF]
Machine learning is quickly becoming an important tool in modern materials design. Where many of its successes are rooted in huge datasets, the most common applications in academic and industrial materials design deal with datasets of at best a few tens of data points. Harnessing the power of machine learning in this context is, therefore, of considerable importance. In this work, we investigate the intricacies introduced by these small datasets. We show that individual data points introduce a significant chance factor in both model training and quality measurement. This chance factor can be mitigated by the introduction of an ensemble-averaged model. This model presents the highest accuracy, while at the same time, it is robust with regard to changing the dataset size. Furthermore, as only a single model instance needs to be stored and evaluated, it provides a highly efficient model for prediction purposes, ideally suited for the practical materials scientist.
In this paper, a new approach based on least squares support vector machines (LS-SVMs) is proposed for solving linear and nonlinear ordinary differential equations (ODEs). The approximate solution is presented in closed form by means of LS-SVMs, whose parameters are adjusted to minimize an appropriate error function. For the linear and nonlinear cases, these parameters are obtained by solving a system of linear and nonlinear equations, respectively. The method is well suited to solving mildly stiff, nonstiff, and singular ODEs with initial and boundary conditions. Numerical results demonstrate the efficiency of the proposed method over existing methods.
This paper proposes an approach based on Least Squares Support Vector Machines (LS-SVMs) for solving second order partial differential equations(PDEs) with variable coefficients. Contrary to most existing techniques, the proposed method provides a closed form approximate solution. The optimal representation of the solution is obtained in the primal–dual setting. The model is built by incorporating the initial/boundary conditions as constraints of an optimization problem. The developed method is well suited for problems involving singular, variable and constant coefficients as well as problems with irregular geometrical domains. Numerical results for linear and nonlinear PDEs demonstrate the efficiency of the proposed method over existing methods.
LS-SVM approximate solution to linear time varying descriptor systems [Automatica 2012][Matlab Code][PDF]
This paper discusses a numerical method based on Least Squares Support Vector Machines (LS-SVMs) for solving linear time varying initial and boundary value problems in Differential Algebraic Equations (DAEs). The method generates a closed form (model-based) approximate solution. The results of numerical experiments on different systems with index from 0 t0 3, are presented and compared with analytic solutions to confirm the validity and applicability of the proposed method.
This paper introduces an estimation method based on Least Squares Support Vector Machines (LS-SVMs) for approximating time-varying as well as constant parameters in deterministic parameter-affine delay differential equations (DDEs). The proposed method reduces the parameter estimation problem to an algebraic optimization problem. Thus, as opposed to conventional approaches, it avoids iterative simulation of the given dynam- ical system and therefore a significant speedup can be achieved in the parameter estima- tion procedure. The solution obtained by the proposed approach can be further utilized for initialization of the conventional nonconvex optimization methods for parameter estima- tion of DDEs. Approximate LS-SVM based models for the state and its derivative are first estimated from the observed data. These estimates are then used for estimation of the unknown parameters of the model. Numerical results are presented and discussed for demonstrating the applicability of the proposed method.
Parameter Estimation for Time Varying Dynamical Systems using Least Squares Support Vector Machines [16th IFAC Symposium on System Identification, 2012][PDF][Github]
This paper develops a new approach based on Least Squares Support Vector Machines (LS-SVMs) for parameter estimation of time invariant as well as time varying dynamical SISO systems. Closed-form approximate models for the state and its derivative are first derived from the observed data by means of LS-SVMs. The time-derivative information is then substituted into the system of ODEs, converting the parameter estimation problem into an algebraic optimization problem. In the case of time invariant systems one can use least-squares to solve the obtained system of algebraic equations. The estimation of time-varying coefficients in SISO models, is obtained by assuming an LS-SVM model for it.
Domain adaptation learning is one of the fundamental research topics in pattern recognition and machine learning. This paper introduces a Regularized Semi-Paired Kernel Canonical Correlation Analysis (RSP-KCCA) formulation for learning a latent space for the domain adaptation problem. The optimization problem is formulated in the primal-dual setting where side information can be readily incorporated through regularization terms. The proposed model learns a joint representation of the data set across different domains by solving a generalized eigenvalue problem or linear system of equations in the dual. The approach is naturally equipped with out-of-sample extension property which plays an important role for model selection. Furthermore, the Nyström approximation technique is used to make the computational issues due to the large size of the matrices involved in the eigen-decomposition feasible. Experimental results are given to illustrate the effectiveness of the proposed approaches on synthetic and real-life datasets.
A software tool SYM-LS-SVM-SOLVER written in Maple is developed to derive the dual system and the dual model representation of LS-SVM based models, symbolically. SYM-LS-SVM-SOLVER constructs the Lagrangian from the given objective function and list of constraints. Afterwards it obtains the KKT (Karush-Kuhn-Tucker) optimality conditions and finally formulates a linear system in terms of the dual variables. The snapshot of the GUI of the developed solver is shown in the following Figure.
