Update, November 28, 2011
The flawed lottery system for the Western States Endurance Run remains in place for the 2012 event. To learn how two recently conducted mock lotteries demonstrate that the group of three-ticket holders has worse odds than either the group of two-ticket holders or the group of one-ticket holders, follow this link.
Update for 2012 Lottery Process
Lottery entry for the 2012 Western States Endurance Run has been open since Saturday, November 12, 2011, and the main problems with the new system have already manifested quite clearly. The following analysis is based on a snapshot of entries from Ultra Signup at 2:15 pm Pacific on November 15. At this time, 1470 people had put their names in the lottery process. Of these, 877 had one ticket in the lottery, 376 people had two tickets, and 217 people had three tickets.
Starting last year, the lottery system changed to grant another ticket for each consecutive year not selected for The Run since 2010. This means that for this sample we have 2280 tickets in the lottery for the 1470 people: 877 for the one-ticket holders, 752 for the two-ticket holders, and 651 for the three-ticket holders. Right away we notice the "dilution effect" whereby the larger pool of two-ticket holders has better odds than the smaller pool of three-ticket holders. This means that the pool of two-ticket holders, as a group, has better odds of getting selected than the pool of three-ticket holders.
This is not a fluke; consistently, since the opening of the lottery, the numbers have worked out this way. It seems a little counter-intuitive, when the individual odds work the other way. Yes, it is true that if you have three tickets your odds are three times better on the first draw than for someone who has one ticket, and two times better if you have two tickets. But these odds are vanishingly small to start with - less than five hundredths of one percent for a person holding one ticket.
To state this differently, the one-ticket holders, as a group, in this scenario, have fewer than 500 chances out of one million of getting selected - very low! What is twice or three times such a small number? Still a very small number. So we are left with multiple-ticket holders persisting with lousy odds and the two-ticket holders, as a group, having better odds than the three-ticket holders.
To paint the picture perhaps more plainly: If we prorated all the tickets in this scenario based on a total ticket pool of 100, we would have 38 tickets for one-ticket holders, 33 for two-ticket holders, and 29 for three-ticket holders - close to evenly trisected. It is clear that the odds of one group over another do not meaningfully differ. Surely we can have a better system than this.
People have asked me why I don't enter the lottery. I have several reasons, one of which is that I have already done the run and many people who want to do the run have not had the opportunity. It also concerns me that some two-ticket holders ran last year, having achieved entry by other means. That does not seem right or fair. It verges on rude, in fact, and unbecoming of the wild public lands that we adore.
The current lottery system is an embarrassment. Let the calls for transparency and more sophistication echo loudly in the halls of the Western States Board. Transparency could begin in two ways: publish the selection algorithm and open it for comment; plainly publish on the Western States website the means of entry for each selected entrant.
About two weeks after the lottery for the 2011 Western States Endurance Run, the following article appeared with an engaged discussion on Bryon Powell's website. The event's website provides details on "automatic" and "special consideration" entrants. As post-lottery entrants continue to be added to the list of participants, the call for transparency and accountability in the lottery process, well stated here, only grows. The article below focuses rather on the insufficiency of the Western States lottery system itself, particularly on the illusion of significantly increased lottery odds for those who receive an extra lottery ticket for each consecutive failure to gain entry. The Western States Board of Trustees hastily substituted an inadequate lottery bias for the prior "two-time loser" (TTL) policy that guaranteed entry to applicants the year after two consecutive failures to gain entry. While it is true that the TTL policy could not continue to work in the context of fixed supply and increasing demand, even at double or triple the current cost of entry, the new system, aside from denying hope to the chronically unlucky, further entrenches the event's reputation for fiat and favor in the entry process, very unbecoming a not-for-profit entity. If not improved, the current system will undermine the event's integrity for many, not just the current few.
I did not enter the lottery for the 2011 Western States run, and I’m not entered otherwise, so I can claim at least some objectivity for the following critique of the lottery process that follows.
For the 2010 run, it became clear that the “two-time loser” policy for getting into the run - not picked in the lottery two years in a row; you’re in the next year, assuming you qualify - had to go; the numbers became impossible in that before very long the entrants would be entirely two-time losers, and even then impossible in the context of the cap imposed by the Forest Service for the number of runners who can pass through the Granite Chief Wilderness at one time. So there had to be a change. So no more two-time losers. Instead, for every year you did not get selected in the lottery, you would get an additional ticket the following year, assuming you qualified and wanted to enter. If you were not selected one year, you would have two tickets next year; if you were not select the second year, you would have three tickets in the following year, and so on.
This process will not work. The odds for people with multiple tickets never improve as much as might seem intuitive. In fact, depending on how many people enter the process for the 2012 run, it could actually happen, and pretty easily, that the odds for the three-tickets holders are worse than their odds the prior year when they held two tickets. The odds for multi-ticket holders will never be 50 percent, or even close, as was reported at the lottery for the 2011 run.
The Western States lottery fits a type of statistical problem that has the scary name hypergeometric distribution. It is like a binomial distribution (itself not a very comforting name) except that there is no “replacement,” which means that if your ticket is drawn, you are removed from the process; in other words, you cannot get selected more than once.
For the 2011 run, there were 2115 tickets in the lottery, 1113 for people who had one ticket and 501 times two for people who entered the process for the 2010 run, did not get selected, and wanted to try their luck again. There were 213 slots available before the “had to be at the lottery” selections. If you plug those numbers into a hypergeometric distribution (Excel makes it pretty easy to do that with the HYPGEOMDIST function), the result is that people who had two tickets had a probability of being selected of just over 18 percent, while people with one ticket had a probability of just over 10 percent. These probabilities are from the point of view of being selected once during the entire process, not the same as the probability at any individual draw, so you might say from the cumulative perspective. Put another way, the probability of a person having two tickets not getting selected at all was about 82 percent, and for a person having one ticket about 90 percent, given 2115 total tickets and 213 spots available.
As the special-consideration entrants continue to come into the run, it is hard to tell now how many one-ticket holders got selected, and how many two-tickets holders, during the lottery process. Of the 1113 one-ticket holders originally listed, 963 are now listed, while of the 501 two-ticket holders originally listed, 400 remain listed. This means 150 from the one-ticket holders and 101 from the two-ticket holders, which means that almost 14 percent of the one-ticket holders got in - by one means or another - and just over 20 percent of the two-ticket holders got in - not too far off the statistical expectation. But the total, 251, is more than the total selected at the lottery, 213. So let’s just guess that 120 of the one-ticket holders were selected in the lottery process while 93 of the two-ticket holders got in. That would put the probability very close to exactly the statistical expectation, which might give us some confidence in the statistical analysis.
So, the probability for the two-ticket holders was not as good as we might have thought, but surely, next year, when the new form of “two-time loser” has a third ticket, the odds go up? Maybe not. The odds could go up, but could easily go down, and in any event remain lower than might intuitively seem to be the case. Let’s say that for the 2012 run we have 1000 one-ticket holders, 700 two-ticket holders, and 300 three-ticket holders, assuming that about two-thirds of the one-ticket holders who did not get in for the 2011 run decide to try for the 2012 run and maybe three-quarters of this year’s two-ticket holders decide to try their luck again. How do those numbers crunch in a hypergeometric distribution? The one-ticket holders would have a probability of a little over 6 percent; the two-tickets holders, a little over 12 percent; and the three-ticket holders, just under 17 percent - worse odds for the three-ticket holders than they had as two-ticket holders. Let’s stretch the guesswork out to include 200 people who hold five tickets for the 2014 run and 250 people who hold four tickets; the odds for the five-ticket holders are about 17 percent - still less than their odds as two-tickets holders. This process does not work.
All numbers aside, I find myself more concerned about the purely human effects involved. In a statistical distribution, there are “outliers” - a few very lucky people and few very unlucky people. I am concerned about what is obviously some pretty significant emotional devastation for some really unlucky people who work themselves to the bone year after year to have a shot at getting into the Western States run and do not get in. This problem is well known, used to be solved by the “two-time loser” policy, does not have a new solution, and is not helped when people are just as competitive about getting into the run as they are about running the event.
I don’t have any bright ideas about how to fix the problem, but I think that we need to open the discussion and, for whatever ideas make the initial cut, apply some probability theory to them and see how they shake out. Let’s not forget that people are involved here, and that the system should be as fair as all the people who care about it have brains to make it fair. One thing that I think would help the larger process is transparency about the selections. Each person in the run should have an annotation about the selection method, such as “got in at the lottery” or “selected by Michigan Bluff aid station” or “Montrail sponsor selection” or “won Miwok” or “the Board feels that this runner will bring a lot of good beneficial publicity to the race.” It does not well serve the Western States run the impression shared by many of a “smoked-filled room” of special consideration. Perhaps people could apply for special consideration, make their case in writing, and be subject to a vote of the applicants, of the people.