Driven by the pandemic, higher education institutions have stopped requiring placement tests when students apply to college. This has led to fewer incoming students providing ACT or SAT scores (as low as 20% at MSU). Given that these scores have been historically used to place students,  this has caused a need for other student success proxies to be used to place students into first-year math and writing courses. This project reviews the accuracy of the current math placement model at Montana State University and then looks to use other models to place students more accurately in courses. The overall goal is to boost student success, retention, and graduation rates, recognizing the crucial role of accurate course placement in a student's educational journey and future success. 

To review the accuracy of the current model, f-scores were calculated by using the following logic: students who are compliant and unsuccessful are considered a false negative, students who are uncompliant and successful are a false positive, students who are compliant and successful are a true positive, and lastly students that are compliant and unsuccessful are a true negative. Based on this logic, the table below shows the corresponding f-scores at each course level. The lowest f-scores are seen in 250 and 290-level courses. An overall f-score for the current model of .90 indicates that the current model is performing well in both precision and recall. It is worth noting that there are many more instances where students were complaint vs. uncompliant, so this measure of f-score may be skewed to a higher score, as false positives and false negatives are factored into the denominator in calculating precision and recall.

Note that for all machine learning models, the tables below depict the f-scores calculated at the course level for each input variable and an overall f-score for each variable and variable combination model explored. These scores show the accuracy of the models holistically and at the course level.

Since the outcome variable for this project is whether a student was successfully placed in a course, the first model explored to improve student placement was a logistic regression. Logistic regressions predict binary outcomes, identifying the relationship between a dependent variable and multiple independent variables. 

Using logistic regression to forecast student success based on variables like placement scores and high school GPA resulted in models that, while statistically significant in some cases, displayed low pseudo-R-squared values across various course levels. This suggests that, despite a certain level of statistical significance indicating a better fit than the null model, the actual explanatory power of the models still needs to be improved. Essentially, the models confirm a relationship between the chosen variables and student success, yet they capture only a tiny fraction of the variance in outcomes. The low pseudo-R-squared values indicate that other unconsidered factors may significantly influence student success or that the relationships between the variables and success still need to be fully captured by logistic regression in its current form.

After reviewing the outcomes of the decision tree, a random forest model was explored. Random forests aggregate many decision trees, which could lead to improved placement. We saw improvement in the model's overall accuracy except when using Ed Ready scores and high school GPA as the input variables. A f-score of .47 was seen in the decision tree model and only .45 in the random forest model. 

 Ed Ready scores alone appeared to have the highest accuracy at .47. However, in reviewing, accuracy in student placement is 0 in 150 and 290. Depending on the course, different variables appear to do better at placing students. For example, at 100 course levels, Ed Ready scores seem more accurate in placing students. In contrast, Ed Ready Scores at the 500 level appear less precise in placing students than other variables. Ed Ready Scores are the most successful in predicting student course placement. Even with improvements to the accuracy of student placement seen in the random forest model, the overall accuracy of student placement is lower than the current success levels seen using the current math placement model.

Lastly, to see if the accuracy of student placement could be further improved, another classification model was used, in this case, XG boost. Like the random forest model, it builds multiple decision trees; however, each tree corrects the errors of the previous tree, and then an algorithm is used to minimize loss when adding new trees. Here, slight improvements to accuracy were seen in all input variables except for Ed Ready scores, where the accuracy stayed the same with an f score of .47. This model appears to have higher levels of accuracy where we see seven f scores above .6. In contrast only 5 of scores above .6 were seen in the random forest output and only 4 in the classification tree output. Even though this is the best statistical model for placing students, the overall placement accuracy is again lower than students' success with the current placement model.