Stacking Classifier

In this project, we implemented a stacking classifier to enhance the prediction performance of our model. Stacking is an ensemble learning technique that combines multiple classifiers (base classifiers) and a final classifier (meta-classifier) to produce a stronger model.  Here’s a detailed explanation of the process and components used in our stacking classifier: 

1）Importing Necessary Libraries

To construct our model, we had to import several libraries from SKlearn specifically designed for the different types of machine learning classifiers available.

2）Defining Base Classifiers:

Linear Support Vector Classifier (LinearSVC): A linear SVM classifier used for both datasets with a large amount of features and for modeling linear relationships between variables .

Calibrated Support Vector Classifier (CalibratedSVC): Converts the output of the Linear Support Vector Classifier into probability estimates using sigmoid calibration. The Calibrated Classifier here uses k-fold cross vlaidation to make sure that the Linear Support Vector Classifier's probability estimates are reliable and less prone to overfitting by training and calibrating on different subsets of the data.

XGBoost Classifier (XGBC): A powerful gradient boosting classifier used for large datasets and known for its performance and efficiency and its ability to model relationships between variables that are more complex than linear. Gradient boosting is a machine learning technique for regression and classification that builds a model using decision trees, and each tree being made in gradient boosting corrects errors of its predecessor[19].

With all that said, our hope was that by combining the strength of both classifiers, we would be able to produce a model that could classify credit scores with high accuracy despite there being a both a large number of data instances and large number of features used inside with different kinds of relationships between them.

3) Defining the Final Classifier:

Logistic Regression: Used as the final estimator to combine the predictions from the base classifiers described above. We thought that the logistic regression was a good choice for the final classifier due to its simplicity, interpretability, and its effectiveness in predicting discrete variable outcomes.

4) Building the Stacking Classifier: 

Once the base and final classifiers classifiers were defined, we decided to use a stacking classifier to the fact that stacking classifiers effectively integrate multiple machine learning algorithms to create a more powerful and accurate predictive model. In our stacking classifier, cross-validation is used to split the our project dataset into multiple folds. For each fold, the models used as based estimators are trained the training portion of our dataset consisting of the remaining K-1 folds. Once that is done, the predictions of the base estimators on the current validation fold are used to train the stacking classifier's final estimator, which combines these predictions to improve the overall classification performance.

Once our model was complete, we decided to train and test our finalized model using cross-validation. By doing so, we were able to both train and test the model on different subsets of our dataset and ensure that every data instance was used for both training and testing at different stages of the evaluation. Doing this allowed us to assess the model's performance more reliably by reducing the risk of overfitting and ensuring that the model generalizes well to new, unseen data. The procedure for doing this went as follows:

 1）Importing Essential Libraries

To be able to use cross-validation to evaluate our finalized model, we had to import several libraries from SKlearn specifically designed for the different types of machine learning classifiers available.

2）Setting Up 10-Fold Cross-Validation

In our project, we have set up KFold cross-validation with 10 splits. For each fold, we had 90% of the data used for training the model and 10% of the data used for testing the model. By using these parameters, we ensured that there would always be plenty of data to train our model and a reasonable amount of data left to test the model throughout the entire KFold cross-validation process. In addition to this, shuffling was added to our model to ensure that each fold is representative of the entire dataset. Without shuffling, there is a risk that the data may contain patterns based on the order, such as temporal or sequential patterns, which could introduce bias into the training and testing process. Hence, shuffling reduces this risk by randomly distributing the data points, ensuring that each fold contains a diverse subset of the data.

3） Initializing Storage for Results

In addition to setting up the 10-Fold Cross-Validation, we have set up a list that would be used to collect the accuracy scores obtained when evaluating our finalized model through each of those folds. 

4） Defining the Path for Saving Results

Once setting up the necessary storage for all the accuracy scores obtained when evaluating the model was done, we also had to declare a path on our computer that would be used as the location where all the confusion matrices obtained when evaluating our finalized model would be saved.

5）Performing the Cross-Validation

Once all the required setup was completed, we went ahead and used our setup KFold cross-validation to evaluate our finalized model. As the model was being evaluated, the related confusion matrices and accuracy scores were stored in the list and computer location designed to store these results so we could go look at them and analyze them later.