Support Vector Machine

Why Use Kernels?

Real-world data are often not neat and messy, so that they are not "linearly" separable; that is, a straight line cannot classify them correctly. SVMs acquire this ability to get around the data not being linearly separable through the kernel trick. The kernel trick is how SVMs can map the data into a higher dimensional space, where the data may be linearly separable. The kernel function produces this higher dimensional coordinate for the data, without actually computing the transformed line in the new coordinate; this advantage lessens the time computing time and amount of data. The kernel trick allows for SVMs to fit complicated patterns and boundaries without keeping the computational cost heavy and high.

Why Is the Dot Product So Critical?

The dot product drives the inner workings of SVMs and also plays an important role in almost all computations in SVMs when kernels are used. In SVMs, the margin and decision boundary are essentially defined by dot products between data points. The kernel trick simply works by substituting the dot product in the SVM with a specific kernel function that represents a dot product in larger dimensional space. This gives the flexibility that SVMs can learn non-linear relationships without actually manipulating larger dimensions of the data. Without both (the dot product and significance of the kernel trick), SVMs would lose too much of their flexibility and power.

What Do the Polynomial and RBF Kernels Look Like?

The polynomial kernel takes the original feature space created by the input features and transforms it to a high-dimensional polynomial space. It also has the ability to model curved decision boundaries of the support vector classifier in many ways by combining input features in different polynomial ways. The polynomial kernel is mathematically written as:

• K(x, x') = (x · x' + r)^d

• where r is a constant term and d is the degree of the polynomial. A degree of 2, for example, introduces squared terms and interactions between features.

The Radial Basis Function (RBF) kernel - also known as the Gaussian kernel - sends data into infinite-dimensional space; it can create incredibly flexible, circular and/or blob-like decision boundaries. The RBF kernel is given by:

• K(x, x') = exp( -gamma * ||x - x'||^2 )

• where gamma controls how tightly the data points influence each other. The RBF kernel is powerful when the decision boundary is very complex and wavy.