S5 The Regression Effect

The LLN creates a Paradox: Future is independent of the Past, but it appears that EXCESSES in the past are BALANCED by DEFICITS in the future to arrive at the AVERAGE. The idea of SWAMPING says that this does not happen -- it is just that there are so many trials on the way to infinity that any amount of imbalance in an initial set of trials will be completely overwhelmed by the events which occur in a much larger set of trials which is to come in the future. This is an important part of the explanation, but there is another important piece which must be understood for a complete explanation. This missing piece is called the REGRESSION EFFECT, which is important in its own right.

REGRESSION EFFECT: Consider a sequence of 10 Flips of a coin in which 9 Heads and 1 Tail was Observed. Considering these as IID Bernoulli RV's by identifying Heads with 1 and Tails with 0, the observed average is 90% but the Expected Value is 50%. What will be the situation after another 10 trials? Consider TWO possible cases for the second set of 10 trials: Event HIGH occurs if 9 or 10 Heads are observed. Event LOW occurs if 0 to 8 Heads are observed. The ratio 90% on the first ten trials will rise or remain the same ONLY when event HIGH occurs. When event LOW occurs, the ratio of Heads will FALL from 90%. The probability of HIGH is 1% while the probability of LOW is 99%. This means that with 99% probability the ratio will fall and with 1% probability the ratio will rise or remain the same (after another 10 trials). So it appears that we can confidently predict that the Current Ratio of Heads, which is EXCESSIVE, will correct itself and move towards a more balanced ratio. Note that this is NOT a SWAMPING effect because we are only considering the next 10 trials.

In other contexts, this is called a REGRESSION effect. This happens because something VERY UNUSUAL happened in the first 10 trials -- the number of HEADS was much larger than normal. The odds of the same unusual thing happening AGAIN are very low, just like they were very low the first time. The chances are good that the next set of 10 trials will give a normal -- average -- number of heads. So if we start out with an unusual outcome, the second one will not be so unusual, and the average of both outcomes will be less exceptional than the first outcome and closer to average. This is called the REGRESSION EFFECT. When it was first used, it was said that if the parents are exceptionally tall -- far above population average -- their children will be closer to the population mean, and not so exceptional. There is REGRESSION towards the mean. The same holds on the other side. If the parents are exceptionally short, the children will be somewhat taller and closer to the population mean. One can think that the childrens height is based on overall population average and parental average. So exceptional parents will tend to have children who are not so exceptional; they will move towards the population mean.

SMALL GAINS and LARGE LOSSES: The above explanation still leaves an unresolved puzzle and apparent paradox about the behavior of the fair coins. Consider the 11th toss. If it comes up HEADS, the ratio 90% will INCREASE. If it comes up TAILS, the ratio 90% will decrease. Both events are equally likely. The fact that there has been a huge excess of HEADS does not make TAILS more likely to come up. Indeed, at ALL steps along the process, Heads and Tails remain EQUALLY LIKELY. So the ratio of heads to tails is JUST AS LIKELY to go up as it is to go down. But how can we RECONCILE this fact, that UP and DOWN moves of the ratio of HEADS are equally likely at ALL STEPS along the process, with the FACT we just showed earlier, that the ratio is OVERWHELMINGLY likely to go DOWN after another 10 STEPS?

To answer this question, look at what happens to the ratio at the 11th step. With 50% probability HEADS will appear, so we will have 10/11=90.9% Heads. With 50% probability TAILS will appear, and we will have 9/11=81.8% Heads. With 50% probability, there will be a gain of about 1% and with 50% probability there will be a loss of about 8%. So even though gains and losses in the NUMBER of heads remain equally likely throughout the process, the ratio of heads does not behave in a symmetric fashion. If we are in a place where there is an excessively large proportion of heads, then further gains are very SMALL, and further losses are very LARGE, although both are equally likely. This explains why, after a number of steps where both gains and losses are equally likely, but gains are always small and losses are always large, the expected change will be in the direction of loss. The proportion of HEADS will come down with very high probability, as already shown directly earlier, in discussing the regression effect.