SVM

INTRODUCTION TO SVMs

Support Vector Machines (SVMs) are powerful supervised learning algorithms used for classification and regression tasks. SVMs aim to find the optimal boundary (called a hyperplane) that separates classes of data points in a dataset. The key idea behind SVMs is to maximize the margin—the distance between the nearest data points of each class and the hyperplane—ensuring better generalization and robustness to unseen data.

WHY SVMs ARE LINEAR NEAR SEPERATORS

SVMs work as linear separators by identifying a hyperplane that divides the data into distinct classes. For linearly separable datasets, this hyperplane can efficiently separate the classes while maximizing the margin between them. For datasets that are not linearly separable, SVMs rely on kernel functions to transform the data into a higher-dimensional space, making it easier to find a linear boundary.

HOW THE KERNEL WORKS

A kernel function transforms the original feature space into a higher-dimensional space, allowing SVMs to classify data that is not linearly separable. The kernel computes the similarity between two data points without explicitly calculating their coordinates in the new space. This is achieved through the kernel trick, which is based on dot products between data points in the transformed space. 

WHY THE DOT PRODUCT IS SO CRTICAL?

The dot product is central to the kernel trick because it measures the similarity between two data points in the transformed feature space, allowing SVMs to compute relationships and boundaries in higher dimensions indirectly, without ever explicitly calculating the transformation. This significantly reduces computational complexity while preserving the ability to classify non-linear patterns effectively.

POLYNOMIAL AND RBF KERNEL FUNCTION

Polynomial Kernel: Captures non-linear relationships by transforming data into higher dimensions.

RBF (Radial Basis Function) Kernel: Maps data into a higher-dimensional space using a Gaussian function, making it effective for complex, non-linear datasets.

EXAMPLE

Polynomial Kernel (r = 1, d = 2)

Let’s compute the similarity between two points x1=(1,2) and x2=(3,4) using the polynomial kernel:

K (x1, x2) = ((1 . 3 + 2 . 4) + 1 ) ^ 2 = ( 3 + 8 + 1 ) ^ 2  = 12 ^ 2 = 144

This demonstrates how the polynomial kernel expands the feature space. 

DATASET PREPERATION

1. Dataset Overview:

The dataset comprises coal shipment records with key attributes such as ash content, heat content, price, quantity, and sulfur content, along with a categorical target variable coalRankDescription. For this analysis, the target variable has been transformed into a binary classification problem, where:

• 1 indicates "Bituminous" coal.

• 0 represents other coal ranks.

This binary classification enables us to apply logistic regression effectively.

2. Binary Target Variable Creation:

The categorical variable coalRankDescription was converted into a binary variable called binary_coal_rank. This transformation simplifies the prediction task for logistic regression.

3. Train-Test Split:

The dataset was split into training (70%) and testing (30%) sets to evaluate the model's generalization ability. This division ensures the model is trained on one subset of the data and tested on a disjoint subset.

INITIAL DATASET

PROCESSED DATASET

TRAIN TEST SPLIT

The dataset was split into training (70%) and testing (30%) sets using the train_test_split function from the sklearn.model_selection module.

Features (X) were standardized using StandardScaler to ensure uniform scaling.

The splitting process was randomized with a fixed random_state value to make results reproducible.

IMPORTANCE OF CREATING A DISJOINT SPLIT

1. Prevention of Overfitting:

• When a model is trained on a dataset, it learns patterns and relationships specific to that data.

• If the same data is used for both training and evaluation, the model's performance might appear artificially high because it has already "seen" the test data during training. This is called data leakage and leads to overfitting.

• A disjoint split ensures that the model is tested on completely unseen data, giving a realistic estimate of how it will perform on new, unseen data in the real world.

2. Replicates Real-World Scenarios:

• In practical applications, models encounter new data that they weren’t trained on. The test set simulates this scenario, providing a realistic measure of model performance.

3. Reliable Performance Metrics:

• Metrics like accuracy, precision, recall, and F1-score calculated on the test set provide unbiased estimates of how the model will perform in real-world use cases.

LINK TO THE DATASET

TRAINING AND TESTING DATA

WHY SVMs REQUIRE LABELED NUMERIC DATA 

Support Vector Machines (SVMs) require labeled numeric data because their mathematical operations, such as calculating distances, dot products, and applying kernel functions, rely on precise numerical representations. As a supervised learning method, SVMs use labels to define clear decision boundaries between classes, enabling effective classification. Additionally, feature scaling, such as normalization or standardization, ensures that all features contribute equally, preventing any one feature from dominating the model's calculations. Kernels like Polynomial and RBF depend on numerical data to map features into higher-dimensional spaces for better class separation. These properties make labeled numeric data essential for the efficiency and accuracy of SVMs.

LINK TO THE CODE

MODELS USED  

1. Linear Kernel SVM

• Performance: Achieved an accuracy of 68.01%. The confusion matrix revealed that while the linear kernel performed reasonably well for the given dataset, it struggled to handle non-linear relationships between features.

• Insights: The linear kernel is best suited for linearly separable data. Its simplicity makes it computationally efficient but limits its ability to capture complex patterns.

2. Polynomial Kernel SVM:

• Performance: With an accuracy of 67.47%, the polynomial kernel slightly underperformed compared to the linear kernel. The confusion matrix indicates that this kernel had difficulty with class overlap, leading to more misclassifications.

• Insights: Polynomial kernels map data to higher dimensions, which is beneficial for non-linear boundaries. However, this kernel requires careful tuning of degree and regularization parameters to achieve optimal performance.

3. RBF Kernel SVM:

• Performance: The RBF kernel achieved the highest accuracy of 69.12%, outperforming both the linear and polynomial kernels. The confusion matrix shows better classification of complex patterns and fewer misclassifications.

• Insights: RBF kernels are powerful for datasets with non-linear relationships, as they map data into infinite-dimensional spaces. The kernel's ability to handle non-linearity made it the best choice for this dataset.

CONFUSION MATRICES FOR C = 1.0 FOR ALL THE KERNELS 

VISUALIZATIONS FOR DIFFERENT C VALUES FOR DIFFERENT KERNELS

FOR C VALUE = 0.1

FOR C VALUE = 10

FOR C VALUE = 100

RESULTS AND OBSERVATIONS

This bar plot illustrates the accuracy comparison of three different kernels used in the Support Vector Machine (SVM) classification: Linear, Polynomial, and RBF (Radial Basis Function).

Linear Kernel: Achieved an accuracy of approximately 68%. This indicates that a linear decision boundary was moderately effective in separating the classes, but it struggled with non-linear relationships in the data.

Polynomial Kernel: Its accuracy was slightly lower, around 67.5%, showing that while it captures more complex decision boundaries compared to the linear kernel, it might have overfit or underfit due to the specific degree used.

RBF Kernel: Performed the best with an accuracy of approximately 69.1%. The RBF kernel is known for its ability to handle non-linear data by transforming it into higher-dimensional space, which likely helped it achieve the highest performance among the three.

The plot highlights that kernel choice significantly impacts SVM performance, with the RBF kernel emerging as the most suitable for this dataset due to its ability to model non-linear relationships effectively.

LEARNINGS

From the analysis and modeling using Support Vector Machines (SVMs), several key insights and predictions emerged that are directly relevant to the carbon intensity of coal shipments:

1. Feature Importance:

   • The models confirmed that features like ash content, sulfur content, and heat content significantly influence carbon intensity levels. This highlights their critical role in determining whether a shipment is categorized as low, medium, or high carbon intensity.

2. Kernel Effectiveness:

   • The RBF kernel proved to be the most effective for this dataset, achieving the highest accuracy (e.g., ~69.12%) by capturing non-linear relationships between features. This suggests that carbon intensity classification involves complex patterns that linear models struggle to capture.

   • The linear kernel, while simpler and faster, was less effective (~68.01%), indicating that linear relationships alone are insufficient for accurately predicting carbon intensity levels.

3. Impact of Regularization:

   • Varying the C values demonstrated how regularization affects model performance. Lower C values (e.g., C = 0.1) resulted in simpler models that generalized well but had slightly reduced accuracy. Higher C values (e.g., C = 100) provided better training accuracy but risked overfitting to noise in the data.

4. Practical Predictions:

   • The SVM models can reliably predict the carbon intensity of future coal shipments, enabling energy companies to optimize shipment processes. For instance, high-carbon shipments can be flagged for review, and adjustments can be made to reduce emissions.

5. Applications in Policy and Operations:

   • These insights are valuable not only for operational efficiency but also for guiding environmental policies. Policymakers can use the predictions to incentivize cleaner coal shipments and develop carbon management strategies.

WHAT DID WE LEARN ABOUT THE TOPIC FROM THE RESULTS?

1. The Role of Kernels in Capturing Non-Linear Patterns

SVMs showcased the importance of kernel selection in achieving high accuracy. By experimenting with Linear, Polynomial, and RBF (Radial Basis Function) kernels:

• Linear Kernel proved effective for datasets with clear linear separability, achieving moderate accuracy.

• Polynomial Kernel added complexity by capturing non-linear relationships, but it struggled with overfitting in some cases.

• RBF Kernel emerged as the most robust, excelling in handling non-linear data distributions and achieving the highest accuracy. This highlights the importance of non-linear transformations in understanding complex relationships among coal shipment features.

2. Feature Interactions and Decision Boundaries

SVMs with the RBF kernel revealed that features like sulfur content and ash content are crucial in distinguishing between high and low carbon-intensity coal. The ability of SVMs to create flexible decision boundaries allowed them to separate overlapping feature distributions, which simpler models like Logistic Regression struggled to address.

3. Impact of Regularization (C Values)

The regularization parameter C played a critical role in tuning the model's performance:

• Lower C values (e.g., 0.1) prioritized a smoother decision boundary but resulted in slightly lower accuracy due to underfitting.

• Higher C values (e.g., 10) improved accuracy but risked overfitting by focusing too much on misclassified points. This balance highlighted how regularization controls the trade-off between bias and variance, making SVMs versatile for datasets with varying complexity.

4. Predictive Capabilities

SVMs proved particularly effective in identifying high carbon-intensity coal shipments. The confusion matrix analysis showed that the RBF kernel minimized false negatives, ensuring that most high-carbon shipments were correctly classified. This is vital for prioritizing environmental interventions and optimizing supply chain decisions.

Concluding Insights

The application of SVMs provided a nuanced understanding of the relationships between coal shipment features and their carbon intensity. By leveraging kernels and regularization, SVMs offered precise classifications and highlighted the importance of non-linear transformations. These results underscore the utility of SVMs in handling complex, real-world datasets, making them a valuable tool for stakeholders aiming to reduce the environmental impact of coal supply chains.