Gain and lift chart

Introduction

Suppose we have a direct marketing campaign problem as follows:

• We want to select a fraction of the population who are more likely to respond, to a marketing campaign.

• We build a model that scores the ad receivers by assigning to each customer the probability that he/she will reply.

• We want to evaluate the performance of this model.

The gain chart indicates the percentage of the overall number of cases in each category "gained" by targeting a percentage of the total number of cases. In other words, we want to see if we wanted to target the samples based on the model's suggestion, how much we could gain.

To better understand this, consider we have two individuals (with not identical features) where both are labeled by 1 (red here). Since they are both labeled by 1, we probably cannot differentiate between these two by saying which one was more likely to be 1. However, a classifier assigns probabilities to the individuals. So, if the probability of one of the two individuals is closer to 1, that case has a greater chance to be classified as 1. That means a model can give us more information since it provides probabilities rather than only binaries 0-1.

Now, we want to use this probabilistic information to segment the samples into groups that are more likely to be 1.

Process of creating the gain chart

Consider doing a process of creating the gain chart as follows, based on the example in the section for tree classifiers.

1. First, we find the probability of each set of features. For instance, for features 1,1,1, (educated, employed, and married), we find out the probability of 1 (being red) is 2/(300+2)=1/151≈0.007. It means for all individuals with these features the model’s prediction of being 1 is 0.007. This indicates that if we set our probability threshold above 0.007 our prediction is 0 and if we set it below this number the prediction is 1. That is how we can find the TPR and Supp for setting the threshold.

2. When we find the probabilities (scores) for all the features, in the next step (table on middle-top) we rank the probabilities from highest to lowest. Then in the next table (below right), we find all samples associated with the probabilities we have found. Now we start setting the threshold probabilities from highest to the lowest and find TPR and Supp and put the values on the next table (far right). Now we have all the pairs and we can draw the points in a figure.

Here, is the gain chart when we do it for our example in the excel sheet plots. In the figure, the points are given by

(TP/(TP+FN),(TP+FP)/(TP+FN+FP+TN)).

The numbers in the denominators are not going to change. For instance, in this example, we have TP+FN=100 and TP+FN+FP+TN=1000. Only the number in the numerators is changing. However, the two numbers TP+FN=100 and TP+FN+FP+TN=1000 are indicating that without any prediction model TP+FN=100 out of the whole TP+FN+FP+TN=1000 will be 1.

Note that as the numbers in the denominator do not change you can just consider this as a comparison between TP and TP+FP.

The procedure is essentially the same as the one that we used to find the gain chart. The only extra thing we need to do is to find TPR/Supp and put them in another column.