Embedded Files
Ensemble Learning: Random Forest and Voting
Ensemble learning combines multiple "weak" models to create a single "strong" model that generalizes better than any individual component. For this project, two ensemble techniques were applied: Random Forest and Voting Classifier.
Random Forest is a bagging method that trains hundreds of Decision Trees on random subsets of the data and random subsets of features, then averages their predictions. This reduces overfitting while maintaining the interpretability and non-linear modeling power of trees.
Voting Classifier combines predictions from multiple distinct model types (e.g., Logistic Regression, Decision Tree, and SVM) by taking a majority vote. This hedges against the weaknesses of any single model.
Results
Random Forest achieved 87.0% accuracy and the Voting Classifier achieved 84.4%. The Random Forest feature importance chart revealed that Region_Enc was overwhelmingly the most important feature, followed by Year, then Military_Exp_Percent_GDP, and finally Period_Enc. This ranking remained consistent across all tree-based models in the project. The Voting Classifier, which combined Logistic Regression, Decision Tree, and SVM, performed marginally better than its individual components by hedging against each model's weaknesses Logistic Regression's linear assumption, Decision Trees' tendency to overfit, and SVMs' sensitivity to outliers.
Comparing all six models tested (Naïve Bayes, Logistic Regression, Decision Tree, SVM, Random Forest, Voting), the tree-based methods consistently outperformed probabilistic and linear models. Random Forest and the single Decision Tree with Entropy criterion tied as the best performers, both exceeding 84% accuracy. This makes intuitive sense: the post-Soviet dataset has categorical features (Region), temporal features (Year), and sharp non-linear boundaries; exactly the conditions where tree-based models excel.
Conclusion:
The ensemble methods cap this project with a clear verdict: no amount of algorithmic sophistication can overcome the fundamental structure of the data. Random Forests and Voting Classifiers are among the most powerful tools in machine learning, yet they improved accuracy by only 1-2 percentage points over a simple Decision Tree. This is not a failure of the methods it is proof that the post-Soviet economic divergence was driven by a small number of highly predictive, easily measured factors that simple models could capture. The strongest predictor, Region, is not something a government can change through policy. The second strongest, Year, represents the inexorable passage of time and global economic cycles beyond any republic's control. Only the third predictor, military burden, was a true policy lever and even there, the data shows it mattered less than geography. The lesson for policymakers is sobering: structural conditions often matter more than decisions.
Page updated
Google Sites
Report abuse