4th Down

Young Sheldon's math is correct. His statistics wrong.

The problem is the contradiction in two statements

"Statistically, always punting on 4th down makes no sense."

"...since they convert on 4th down 50% of the time."

The statistical problem is that we cannot know the Aggies conversion probability on 4th down if they always punt on 4th down. We cannot observe the counter-factual outcome. Even if they don't always punt, they punt a lot. This should make us very wary of using numbers from the times they do actually go for it.

Expected Points

In Sheldon's discussion he uses probability of a score. We will use expected points. These are calculated by breaking the field into a set of states and solving the dynamic programming problem. Luckily, we don't actually need to do this. Rather, we can simply download the data from https://github.com/maksimhorowitz/nflscrapR. See chart below.

To determine the value of going for it on 4th down we need to know a few things. What is the expected points from punting the ball? What is the expected points from successfully getting first down? What is the expected points from turning the ball over on downs? Finally, what is the probability of successfully getting first down?

In algebra, the Aggies should go for it in 4th down if the following inequality holds.

p E(P | Y = 92) + (1 - p)(- E(P | Y = 5)) > -E(P | Y=40)

Let p is the probability of successfully making first down. Sheldon claims this number is 0.5. E(P | Y = 92) means the expected points when the ball is 92 yards from your goal line. I assume that the Aggies are able to move the ball 3 yards and gain first down. The ball is then on their "own 8". Note that when you don't have the ball, you get negative expected points. So if you punt the ball to your own 40 yard line, the other team has 40 yards to their goal line.

Using the chart below we can replace the algebra with values and determine the probability needed to make going for it the best option.

q (-1.8) + (1-q)(-4.5) > -2.8

q > 1.7/2.7 = 0.63

OK. So Sheldon's math is not correct either. The Aggies should punt if their probability of converting is greater than 0.63. Of course this is NFL, not Texas college football from the 80s.

What is the Probability of Converting?

Is the probability of converting greater than 0.63? We don't know. We can't actually see the probability of converting on 4th down when teams don't actually go for it on 4th down. The Berkeley macroeconomist, David Romer, suggests than we can use 3rd down instead.

Romer correctly points out that the change in payoffs from third down to fourth down will lead the Offense and Defense to change their strategies. However, he incorrectly argues that these changes would offset each other. We cannot use 3rd down data to estimate the success probabilities. But we may be able to use 3rd down data to estimate the game and model what would happen if the Aggies went for it on 4th down.

Mixed Strategy Nash Equilibrium

John Nash, almost certainly did not say anything like what was suggested in the movie, A Beautiful Mind. For one, his "solution" is not a Nash equilibrium. I'm pretty sure Nash would recognize a Nash equilibrium if it walked into a bar. What he did do, was show that for almost any "game" there exists at least one solution. At least one Nash equilibrium, although it may be in "mixed" strategies.

Below, I create two simple representations of the Third Down Game and the Fourth Down Game. These are zero-sum games. The payoffs are written out for the Offense with the Defense receiving the negative. The strategy Run and Pass for Defense means that they will play defenses that key on either the run or pass. For example, replacing a line backer with a defensive back is associated with pass defense. The two games are identical except that the payoff from failure changes a lot from Third Down to Fourth Down.

Third Down Game

Offense

Defense Run Pass

Run q_rr E(P | Y=92) - (1 - q_rr) E(P | Y=40) q_pr E(P | Y=80) - (1 - q_pr) E(P | Y=40)

-q_rr 1.8 + (1 - q_rr) 2.8 -q_pr 0.5 + (1 - q_pr) 2.8

Pass q_rp E(P | Y=92) - (1 - q_rp) E(P | Y=40) q_pp E(P | Y=80) - (1 - q_pp) E(P | Y=40)

-q_rp 1.8 + (1 - q_rp) 2.8 -q_pp 0.5 + (1 - q_pp) 2.8

q_rr - probability of success given Offense plays Run and Defense plays Run.

q_rp - probability of success given Offense plays Run and Defense plays Pass.

q_pr - probability of success given Offense plays Pass and Defense plays Run.

q_pp - probability of success given Offense plays Pass and Defense plays Pass.

Fourth Down Game

Offense

Defense Run Pass

Run q_rr E(P | Y=92) - (1 - q_rr) E(P | Y=5) q_pr E(P | Y=80) - (1 - q_pr) E(P | Y=5)

-q_rr 1.8 + (1 - q_rr) 4.5 -q_pr 0.5 + (1 - q_pr) 4.5

Pass q_rp E(P | Y=92) - (1 - q_rp) E(P | Y=5) q_pp E(P | Y=80) - (1 - q_pp) E(P | Y=5)

-q_rp 1.8 + (1 - q_rp) 4.5 -q_pp 0.5 + (1 - q_pp) 4.5

Game theory predicts that the outcome will be the Nash equilibrium. As this is in mixed strategies, the equilibrium is where the Offense chooses the probability of Pass such that the Defense is indifferent between Pass and Run. Similarly, for the Defense.

Now 4 Some Econometrics

Note that the probability we are interested depends on the strategies of the Offense and Defense. If both plays Pass, then the probability of success is q_pp. In order to determine the probability of success we need to determine the probability that the Offense plays pass and the probability that the Defense plays pass. We can do this using data on Third down games. From the data we actually observe whether the Offense plays Pass or Run. Unfortunately, we don't really observe what the Defense does. But we can assume the observed strategies of the Offense and the observed success probabilities are the outcome of a mixed strategy Nash equilibrium. In particular, the game written above. Given this assumption, we can estimate q_pp, q_pr, q_rp, q_rr for different locations on the field and distances to go using General Method of Moments.

Some Results

Using NFL data and the approach laid out above, we can model Young Sheldon's example. There is between 1.5 and 3.5 expected points advantage to punting over going for it on 4th down. Sheldon is wrong. The statistics clearly state that the Aggies should punt.

More generally we can ask how the model predictions compare to the actual behavior of NFL coaches. David Romer claims that his model predicts teams should go for it than they actually do. This analysis suggests that NFL coaches should go for it about the same as they actually do.