Embedded Files
Overview
Linear regression models relationship between one or more input features and a continuous numerical output by fitting a straight line (or hyperplane) that minimizes the sum of squared differences between predicted and actual values. The output can theoretically be any real number positive, negative, or fractional. It is used when the goal is to predict a quantity, such as GDP per capita or military spending level. The core assumption is that the relationship between input and output is approximately linear.
A logistic regression is a classification algorithm that predicts the probability that an observation belongs to one of two classes. Despite its name, it does not predict a continuous number instead, it outputs a value strictly between 0 and 1, which is then thresholded (typically at 0.5) to assign a class label. It works by fitting a linear combination of features and then squashing the result through the Sigmoid function to constrain the output to a valid probability range. It is one of the most widely used binary classification models in statistics and machine learning.
Both linear and logistic regression model the relationship between input features and an output using a linear combination of weighted features, and both are trained by optimizing a loss function. The key difference is in the output: linear regression predicts an unbounded continuous value, while logistic regression predicts a probability between 0 and 1 for a discrete class. Linear regression is optimized using least squares (minimizing squared error), while logistic regression is optimized using maximum likelihood estimation. Linear regression is inappropriate for classification because its predictions can exceed the valid 0–1 probability range.
Note on Logistic Regression:
The Sigmoid function is the core transformation that makes logistic regression work as a classifier. After computing the linear combination of features (z = w₀ + w₁x₁ + w₂x₂ + ...), the Sigmoid function maps z to a probability between 0 and 1 using the formula σ(z) = 1 / (1 + e^(−z)). When z is very large, σ(z) approaches 1; when z is very negative, σ(z) approaches 0. This S-shaped curve is what allows the model to express confidence values near 0.5 indicate uncertainty, while values near 0 or 1 indicate high confidence.
How MLE is connected to logistic regression:
Maximum Likelihood Estimation (MLE) is the optimization method used to train logistic regression. The goal is to find the set of weights that makes the observed training labels most probable given the model's predictions. For each training observation, the model outputs a probability; MLE multiplies these probabilities together across all observations and finds the weights that maximize the resulting joint likelihood. In practice, we minimize the negative log-likelihood (also called binary cross-entropy loss) because it is mathematically equivalent and numerically more stable. Gradient descent is then used to iteratively adjust the weights toward this maximum.
Data Prep:
Regression Data Preparation
Logistic Regression requires the data to be scaled because it is sensitive to the magnitude of feature values. Unlike Decision Trees, which split on thresholds, Logistic Regression computes a weighted sum of features meaning a feature with values in the thousands (GDP) would dominate a feature with values between 0 and 1 (Military %) without scaling. StandardScaler was applied to the training set and the same fitted scaler was used to transform the test set, ensuring no data leakage. The binary label (High Performer = 1, Low Performer = 0) was already appropriate for logistic regression, which requires exactly two classes.
The same labeled dataset constructed for Naïve Bayes is reused here, consisting of four features (Military Expenditure %, Year, Encoded Period, and Encoded Region) and a binary label (High Performer vs. Low Performer). The data was split into an 80% Training Set and a 20% Test Set using train_test_split with random_state=42 to ensure reproducibility. These sets must remain disjoint, if the model were evaluated on data it trained on, the accuracy metric would reflect memorization rather than genuine predictive ability, rendering it scientifically meaningless.
Why Train-Test Split?
Every supervised model in this project was trained on 80% of the data and evaluated on the remaining 20%, using train_test_split(X, y, test_size=0.2, random_state=42). The two sets must remain completely disjoint meaning no observation can appear in both sets simultaneously. If the model were allowed to evaluate itself on data it already learned from, the resulting accuracy would reflect memorization rather than generalization to unseen data, making it scientifically meaningless. Setting random_state=42 ensures the exact same split is reproducible by anyone attempting to replicate the results. The training set is used exclusively to fit model parameters, while the test set acts as a simulation of real-world unseen data.
Link to Data
https://github.com/rileythejones/-CUBoulder-DS-CSCI-5612-Project/tree/main/Project%20Module%20Three
Code
All three classification models Logistic Regression, Multinomial Naïve Bayes, and Decision Tree were implemented in Python using scikit-learn and evaluated on the same 80/20 train-test split for a direct apples-to-apples comparison. Logistic Regression was fit on StandardScaler-normalized data, as it is sensitive to feature magnitude. Multinomial NB was fit on a non-negative shifted version of the raw features. The Decision Tree (Entropy, depth=4) was fit on the unscaled feature matrix, as tree-based models are scale-invariant. Confusion matrices and accuracy scores were computed for all three models against the same held-out test set, and results were visualized side-by-side in both a confusion matrix panel and a bar chart summary.
Link to Code
Results
The Decision Tree consistently outperformed both Logistic Regression and Naïve Bayes on this dataset. This is expected given the nature of the data: the relationship between Region, Period, and economic performance is not linear — it is hierarchical and rule-based, which is precisely the kind of structure Decision Trees are designed to exploit. Logistic Regression assumes a linear decision boundary in feature space, which is a restrictive assumption when a categorical feature like Region creates sharp, non-linear boundaries between classes. Naïve Bayes makes the additional restrictive assumption of feature independence, which is violated here since Region and Period are strongly correlated. The Decision Tree's ability to capture non-linear, hierarchical interactions makes it the best-performing model for this project.
Model // Accuracy:
Logistic Regression
76.62%
Multinomial NB
67.53%
Decision Tree (Entropy, d=4)
84.42%
Conclusion
The "Regression" here analysis confirms that the post-Soviet economic transition was fundamentally a non-linear classification problem. Logistic Regression, while a powerful and widely used model, is constrained by its assumption of a linear decision boundary an assumption that does not hold when geographic region creates sharp, discrete separations between successful and struggling economies. The Sigmoid function at the heart of logistic regression is excellent at capturing gradual probabilistic transitions, but the Soviet collapse produced abrupt, structural dividing lines between the Baltic reformers and the rest of the bloc that a linear boundary cannot fully capture. Of the three models, the Decision Tree best represents the actual decision logic of the transition: first, where was the country? Then, when was it? And finally, how much did it spend on its military? These sequential, rule-based questions mirror exactly how historians have always narrated the collapse and the fact that a machine discovered the same hierarchy from data alone is the central finding of this project.
Page updated
Google Sites
Report abuse