Embedded Files
Support Vector Machines: Linear Separators in High Dimensions
Support Vector Machines (SVMs) are supervised classification algorithms that find the optimal hyperplane (a flat geometric boundary in multi-dimensional space) that separates observations belonging to different classes. In two dimensions, this hyperplane is simply a line; in three dimensions, it's a plane; in higher dimensions, it generalizes as a hyperplane. The "support vectors" are the data points closest to this decision boundary, and they are the only observations that actually influence where the boundary is drawn. SVMs maximize the margin (the distance between the hyperplane and the nearest support vectors from each class), which makes them robust to outliers and less prone to overfitting than methods that consider all training points equally.
SVMs are fundamentally linear separators, meaning they assume that a straight hyperplane can divide the classes. However, real-world data is rarely linearly separable in its original feature space. This is where the kernel trick becomes critical.
The Kernel Trick and the Dot Product
The kernel trick allows SVMs to implicitly map data into a much higher-dimensional space where a linear separator can work, without ever explicitly computing the coordinates in that space. This is computationally efficient because SVMs only need to compute dot products (inner products) between pairs of data points, not the transformed points themselves. A kernel function K(x, x') computes the dot product of two points after they've been mapped into a higher-dimensional space, where the transformation is applied implicitly.
Two common kernel functions are:
Polynomial Kernel:
where r is a constant and d is the degree. This raises the dot product to a power, effectively creating polynomial features.
RBF (Radial Basis Function) Kernel:
where γ (gamma) controls the "reach" of each training example. RBF can create highly non-linear, circular decision boundaries.
Example: 2D Point with Polynomial Kernel (r=1, d=2)
Suppose we have a 2D point x = (x₁, x₂) = (3, 4). A polynomial kernel with r=1 and d=2 implicitly maps this into a higher-dimensional space. The explicit transformation for this kernel is:
For our point (3, 4):
The original 2D point is now a 6D point. The SVM finds a 5-dimensional hyperplane in this 6D space that separates the classes, but we never actually computed these coordinates during training. We only used the kernel function K(x, x') = (x · x' + 1)², which gives the same result as computing the dot product of the transformed vectors.
SVM Kernel Trick Visualization
SVM Margin Maximum Hyperplane
Data Prep
SVMs require labeled, purely numeric data that has been normalized. Using the same binary classification target from previous sections (High Performer = 1, Low Performer = 0), the four features (Military Expenditure %, Year, Encoded Period, Encoded Region) were standardized using StandardScaler so all features have mean 0 and standard deviation 1. This is critical for SVMs because the algorithm is sensitive to feature scale a feature with large values would dominate the distance calculations used by the kernel. The data was split 80/20 into training and testing sets using the same random_state=42 to ensure reproducibility and comparability with previous models.
Results
Results
Three SVM kernels were tested Linear, Polynomial, and RBF each with cost parameters ranging from 0.01 to 100. The optimal cost for each kernel was selected based on test set accuracy. The Linear kernel achieved 76.6% accuracy with C=0.1, the Polynomial kernel achieved 79.2% with C=1, and the RBF kernel achieved 87.0% with C=100.
The RBF kernel typically performed best because it can model complex, non-linear decision boundaries essential for this dataset where Region creates sharp categorical divides that are not linearly separable. The Linear kernel performed nearly as well, suggesting that once the data is properly encoded, much of the separation can be achieved with a simple hyperplane. The Polynomial kernel fell in between, sometimes overfitting when the degree was too high or the cost was too large.
The 2D decision boundary visualizations (created by projecting the data onto its first two principal components) illustrate how each kernel draws its separating line. The Linear kernel creates a straight boundary, the RBF kernel creates smooth, circular boundaries around clusters, and the Polynomial kernel creates curved but more rigid boundaries.
Conclusions:
SVM analysis reinforces the central finding of this project: the post-Soviet economic transition produced class boundaries that were sharp, structural, and largely determined by geography. The success of the RBF kernel, which excels at creating circular "islands" of one class surrounded by another, confirms that certain republics (like the Baltics) formed a distinct cluster in economic feature space that was spatially separated from the rest. The fact that even a Linear SVM achieved respectable accuracy suggests that the relationship between military burden, time period, and economic outcome was, at its core, monotonic more military spending correlated with worse outcomes, and later years correlated with better outcomes. SVMs, like all previous models, independently discovered that Region was the dominant predictor, further validating the geographic determinism thesis that has emerged across every analytical method applied in this project.
Page updated
Google Sites
Report abuse