Embedded Files
Overview
Decision Trees are supervised machine learning models that learn a series of yes/no questions about the data in order to classify observations into categories. Starting from a single "root" node, the algorithm recursively splits the dataset into smaller and smaller subsets based on the feature that best separates the classes, until it reaches a terminal "leaf" node that assigns a final prediction. Decision Trees are one of the most interpretable models in machine learning because the resulting structure can be read like a flowchart making them useful not just for prediction, but for explanation.
Decision Trees can be applied to both classification tasks (predicting a category, such as "High Performer" vs. "Low Performer") and regression tasks (predicting a continuous number). They require minimal data preprocessing, handle both numerical and categorical features naturally, and are the foundation for more powerful ensemble methods like Random Forests and Gradient Boosting.
GINI Impurity, Entropy, and Information Gain
At every node, the algorithm must decide which feature and which threshold creates the "purest" split meaning child nodes that contain mostly one class rather than a mix. Two common measures of impurity are GINI and Entropy.
GINI Impurity measures the probability that a randomly chosen element from a node would be incorrectly classified if it were randomly labeled according to the class distribution in that node. A perfectly pure node has a GINI of 0.
Entropy measures disorder or uncertainty in a node, borrowed from information theory. Like GINI, a perfectly pure node has an Entropy of 0.
Information Gain is the reduction in impurity achieved by a particular split. The algorithm always chooses the split that produces the highest Information Gain.
Small Worked Example using GINI:
Suppose a node contains 10 observations: 7 "High Performers" and 3 "Low Performers."
A positive Information Gain of 0.18 means this was a useful split that increased the purity of the resulting nodes.
Why is it possible to create an infinite number of trees?
Because at every node, any feature can be selected as the split variable, and any value along a continuous feature (like GDP per capita or Year) can serve as the threshold. With even a handful of continuous features, the number of possible threshold combinations is theoretically infinite. Beyond that, hyperparameters like maximum tree depth, minimum samples required at a leaf, the impurity criterion (GINI vs. Entropy), and whether features are allowed to be reused all multiply the total number of valid trees that can be constructed from the same dataset. This is precisely why constraining trees with a maximum depth or minimum leaf size is critical without such limits, a Decision Tree will grow until it memorizes the training data perfectly, a phenomenon known as overfitting.
Data Prep
The same labeled dataset constructed for Naïve Bayes is reused here, consisting of four features (Military Expenditure %, Year, Encoded Period, and Encoded Region) and a binary label (High Performer vs. Low Performer). The data was split into an 80% Training Set and a 20% Test Set using train_test_split with random_state=42 to ensure reproducibility. These sets must remain disjoint, if the model were evaluated on data it trained on, the accuracy metric would reflect memorization rather than genuine predictive ability, rendering it scientifically meaningless.
Why Train-Test Split?
Every supervised model in this project was trained on 80% of the data and evaluated on the remaining 20%, using train_test_split(X, y, test_size=0.2, random_state=42). The two sets must remain completely disjoint meaning no observation can appear in both sets simultaneously. If the model were allowed to evaluate itself on data it already learned from, the resulting accuracy would reflect memorization rather than generalization to unseen data, making it scientifically meaningless. Setting random_state=42 ensures the exact same split is reproducible by anyone attempting to replicate the results. The training set is used exclusively to fit model parameters, while the test set acts as a simulation of real-world unseen data.
Link to Data
https://github.com/rileythejones/-CUBoulder-DS-CSCI-5612-Project/tree/main/Project%20Module%20Three
Code: Three DT Models Compared
The assignment requires three trees with different root nodes. We force this by changing the max_features, max_depth, and criterion parameters, which causes the algorithm to discover different optimal splits.
Link to Code
https://github.com/rileythejones/-CUBoulder-DS-CSCI-5612-Project/tree/main/Project%20Module%20Three
Results
Three Decision Trees were trained using different configurations to demonstrate how parameter choices affect the root node and overall structure. Tree 1 used GINI impurity with a maximum depth of 3, Tree 2 used Entropy with a depth of 4, and Tree 3 deliberately excluded the dominant root feature to force the algorithm to find the next most informative split. The confusion matrices and accuracies are shown above. Decision Trees tend to match or slightly exceed Naïve Bayes accuracy on this dataset because they can capture non-linear interactions between features (for example, the combination of being in the Caucasus region AND having a high military burden is jointly worse than either factor alone). The deeper trees (depth=4) typically show higher training accuracy but can begin to overfit, which is reflected in only marginal test accuracy improvements over the shallower tree.
Interpretation:
Tree 1 (GINI, depth=3) achieved 81.82% accuracy and is the most interpretable, with only three levels of splits. Tree 2 (Entropy, depth=4) achieved the highest accuracy at 84.42% by allowing one additional level of splitting, capturing more nuanced interactions between Year and Military burden below the Region root. Tree 3, which was forced to root on Year instead of Region, dropped to 68.83% a 15-point decline that quantifies exactly how much predictive power Region alone contributes to the model. The fact that both Tree 1 and Tree 2 independently selected Region_Enc as the root node, despite using different impurity criteria, confirms that this feature is robustly the most informative variable in the dataset.
Confusion Matrices
Conclusions
Decision Tree modeling produced the strongest predictive results of any method applied in this project thus far. Tree 1 (GINI, depth=3) achieved 81.82% accuracy, Tree 2 (Entropy, depth=4) achieved 84.42% accuracy, and both trees independently selected Region_Ednc (geographic region) as the root node, which is the single most important split. This is a statistically meaningful outcome: without being told anything about Soviet history, the algorithm determined that geography was the dominant predictor of post-Soviet economic fate. Tree 3, which was deliberately forced to exclude Region and instead rooted on Year, dropped to 68.83% accuracy a significant 15-point decline. This controlled experiment is arguably the most insightful result in the entire project. It quantifies exactly how much predictive power comes from regional identity alone versus all other factors: roughly 15 percentage points of accuracy, which is enormous in a binary classification setting. The implication for the topic is clear the economic fate of a post-Soviet republic was largely sealed by geography before a single policy decision was ever made.
Page updated
Google Sites
Report abuse