Rare Event Clustering function in R: Overview

While Working on a project, we discussed with the client on  how to be able to predict fraud. The fraudulent cases come, luckily, only far and between,  only a few fraud cases appear in a long series of control. In terms of Fraud prediction, this was a problem for the team who struggled to implement classical predictive modeling techniques. Fraud detection usually deals with « rare » events phenomena, where we  only have a few cases of fraud (« rare » event) that can be used for the Analysis. 

We worked out an approach based on combined  matrix reduction techniques and clustering that aims at finding homogenous groups (clusters) of observations (cases) within a dataset that contain the highest proportion of  identified « rare » cases.  I refined and coded the all approach in R (with a shiny application) in order to easily and quickly reproduce it on any kind of data.


Unlike predictive methods that purposely look for a relationship between the inputs and a chosen target to produce parameter estimates and model this relationship, the method proposed here is unsupervised and exposes the data to many clustering model trials, maximizing the chance that a cluster is found that regroups a significant number of these "rare" cases. We let the data  « speak » and « reveal » the structure of the relationships within the data that will form natural groups. We then check any potential interesting association by exposing the « rare » events  to the clustering model by cross-tabulating these rare events with the groups in the cluster model. 


https://sites.google.com/site/jcpanalytics/rare-event-clustering/image1_rare_event.png


Test Case using the “Lending Club Loan Data” (Kaggle.com)


The Data: The “Lending Club Loan Data” downloaded from Kaggle.com contain all loans issued through the 2007-2015, including the current loan status (Current, Late, Fully Paid, etc.) and latest payment information. Additional features include credit scores, number of finance inquiries, address including zip codes, and state, and collections among others. The file is a matrix of about 887000 observations and 75 variables.


The “rare” Event: is defined as a  loan for which the “loan status” is "Does not meet the credit policy. Status: Charged Off". Out of the 887000 cases (loans) in the data  we have defined  760 cases as  “rare” events. 


Training and Test set: We have split the data into training and test set (“holdout” sample that will help us “honestly” assess the relevance of the clustering model on unseen data). The training set used to build the clustering models is a random selection of 133336 cases containing 572 event cases. The test data set has 89228 cases containing 188 event cases. All event cases from the source dataset have been picked and randomly split between test and training data.


The modelling: Around 50 different  clustering models will be carried out.  I set the number of clusters parameter of the function to  create a maximum of 25 clusters.  After rejection of poor quality or too correlated variables and selecting the variables that showed the greatest difference between the mean of normal cases and “rare cases”, we ended up with 23 variables as input for the analysis:


"loan_amnt"                  "int_rate"                   "installment"      "annual_inc"     "dti"                        "delinq_2yrs"                "inq_last_6mths"             "open_acc"                   "pub_rec"                   "revol_bal"                  "total_acc"                  "out_prncp"                  "total_pymnt"                "total_rec_int"   "total_rec_late_fee"         "recoveries"               "last_pymnt_amnt"            "collections_12_mths_ex_med"  "tot_coll_amt"       "tot_cur_bal"                "total_rev_hi_lim"           "issue_d"                    "days_since_early_crdt" 

   

The running time to create the 51 clustering models on the training data was 1h15mins.


Comparison of  the  clusters across the different models:  the function produces a table that will show a list the best clusters  across all clustering models tried.

Clusters are sorted in terms of specificity and sensitivity. 

This table will help the analyst  choose the best cluster model (method) according to his own context and purposes.





We will now apply the  “fa_mccam_r”  cluster model on the Test data and check if the sensitivity of cluster 1 holds true.


Scoring the Test data: The shiny application will upload all parameters necessary to  assign each new case from the test set to its closest cluster from the selected clustering model:


1.New cases are projected on the axis of the selected Factorial Axes  that were defined on the training data.
2.These projections will then be used to assign each new case to the closest cluster centroid  from the selected cluster model defined on the training data.  


Do we hold the same sensitivity per cluster  as in the training data ?


If we look at the actual distribution of  rare events across the clusters on the test set ,we see that cluster 1 captures  182 rare cases out of 188 in total, that is a sensitivity of 96,8%. Cluster 2 has 6 rare cases (sensitivity of 3,1%).


Using the clustering model, we identified  a cluster (only 6.8% of the total number of observations) which contains 96,8% of the "rare" event cases we want to capture.

 

Concretely it means that thanks to the cluster model we can zoom in on the test data and focus on 6.8% of the data to capture 96% of the "rare" cases.


The Clustering model is robust and performs as expected (as specified in the modelling) on test data.

We can apply the clustering model on refreshed data to better target /identify rare cases.  


The SHINY application for rare Event scoring (screenshots):



Comments