A few weeks ago, the NCAA began making public a great deal of prior season RPI-related information, including actual ratings.  In addition, due to nc-soccer’s discovery of a problem in the NCAA’s end-of-regular-season RPI ranking report, involving the teams’ listed win-loss-tie records, the NCAA provided me with its Team Sheets and Nitty Gritty report for the 2011 season.  These were pieces of information provided to the Women’s Soccer Committee for their Tournament bracket formation process and the NCAA staff provided them to me to demonstrate that the win-loss-tie records provided to the Committee were correct.  The Team Sheets, among other things, included teams’ Adjusted RPI ratings for the 2011 season.  (According to the NCAA staff, both of these documents now are public documents that will be released on line in about a month.)

With this new information in hand, I have been able to determine that the Adjusted RPI ratings and rankings that the NCAA staff provided to the Division I Women’s Soccer Committee for both the 2010 and 2011 seasons, for use in the Committee’s NCAA Tournament at large selections and seeding, in fact were incorrect.  This appears to be due to a programming error that the NCAA staff made in the middle of the 2010 season and that carried forward into the 2011 season.

The only possible way I could be wrong is if the Women’s Soccer Committee, prior to the 2010 season, authorized significant changes from the bonus and penalty adjustments used in previous seasons.  I have been able to find no record of the Committee authorizing significant changes, nor have I found any record of the Championships/Sports Management Cabinet having approved changes, which I believe would have been required.  Further, I asked a Committee member about whether the Committee had approved any changes and the member had “no knowledge regarding any changes.”  In addition, I have communicated what happened (set out below) to the NCAA staff and to the Women’s Soccer Committee and have had no response from them, including no claim that the Committee approved the changes.

Here is the explanation (slightly modified) that I provided to the NCAA staff and the Women’s Soccer Committee of the error, how it occurred, how it affected the end-of-season rankings, and how the NCAA could avoid similar errors in the future.

What Was the Error?

As you know, the NCAA first computes the “Normal RPI” (I call it the unadjusted RPI) and then computes the Adjusted RPI.  The adjustments are bonuses for good wins and ties and penalties for poor losses and ties.  The error was in the amounts of the bonus and penalty adjustments.

In 2009, the NCAA’s RPI formula used the following bonus and penalty amounts:

I can be certain that the NCAA used the above amounts in 2009 because, using my RPI computation system, they produce adjustment totals going from the unadjusted to the adjusted RPI that in every case match the NCAA’s adjustment totals.  This would not be possible unless these are the right amounts.

In 2010, the NCAA started out using these same bonus and penalty amounts, which were the basis for the first RPI report it issued on October 5, 2010.  I know this because for that RPI report, my RPI rankings matched the NCAA’s exactly.

Some time during the week after releasing the October 5 RPI report, the NCAA staff mistakenly changed the bonus and penalty amounts.  As a result, the computations for the subsequent reports were based on different bonus and penalty amounts.  These changed – and erroneous – bonus and penalty amounts were as follows:

I know these were the amounts the NCAA changed to because, just as for 2009, when programmed into my RPI computation system, they produce adjustment totals going from the unadjusted to the adjusted RPI that in every case match the NCAA’s adjustment totals.  This would not be possible unless these are the right amounts.  As an extra precaution, however, I also tested the above adjustment amounts against the data underlying the NCAA’s October 5, 2010 RPI report, to be sure I am right that these adjustment amounts came into play only after the NCAA issued that report.  I am right:  These new adjustment amounts do not produce the rankings the NCAA had in its October 5 report.  Rather, only the 2009 adjustment amounts produce those rankings.  Thus in mid-season 2010, the NCAA staff changed the adjustment amounts to those in the table immediately above.

If you will look at the amounts of the bonus and penalty awards in relation to the categories and compare them to the 2009 amounts, you will see that this represented a major change in the overall structure of how values are awarded.  In particular:

Bonus amounts for wins against the top 40 teams went from 0.0032 (away) – 0.0030 (neutral) – 0.0028 (home) to 0.0026 (away) – 0.0024 (neutral) – 0.0022 (home), thus significantly reducing the bonuses for good wins.

Bonus amounts for wins against teams 41-80, in the correct 2009 formula, were 0.0018 (away) – 0.0016 (neutral) – 0.0014 (home), significantly less than the bonus amounts for wins against top 40 teams.  In the incorrect 2010 formula, however, the amounts for wins against teams 41-80 were 0.0020 (away) – 0.0018 (neutral) – 0.0015 (ties), only slightly less than the bonus amounts for wins against top 40 teams.  Thus the 2010 error changed the overall structure of the bonus amounts for wins from one in 2009 that strongly emphasized wins against teams in the top 40 to one that only slightly favored wins against teams in the top 40.

Bonus amounts for ties against teams in the top 40, in the correct 2009 formula, overlapped and were almost the same as the bonus amounts for wins against teams in the 41-80 range.  On the other hand, in the incorrect 2010 formula, the bonus amounts for ties against teams in the top 40 all were less than all of the bonus amounts for wins against teams in the 41-80 range.  Thus here too, the 2010 error deleted the correct 2009 formula’s strong emphasis on good results, in this case ties, against teams in the top 40 range.

In a further but less significant error, the new formula, rather than imposing a 0.0002 penalty for away ties against teams ranked 135-205 as it clearly should have, mistakenly awarded a 0.0002 bonus.

In 2011, the NCAA continued with the incorrect mid-stream 2010 amounts, but with a correction from a bonus to a penalty for away ties against teams rated 135-205, and with the amounts of the awards slightly adjusted.  Notwithstanding the slight adjustments, however, the basic format of the 2011 awards matched the format of the 2010 awards, thus continuing the elimination of the correct 2009 and early 2010 heavy emphasis on wins and ties against teams ranked 1-40.  The 2011 amounts were as follows:

Here too, I know these are the amounts the NCAA used in 2011 because when programmed into my RPI computation system, they produce adjustment totals going from the unadjusted to the adjusted RPI that in every case match the NCAA’s adjustment totals.

The NCAA staff does not have the authority to make significant changes in the overall structure of the bonus and penalty amounts.  All such changes are subject to the approval of the Women’s Soccer Committee (and I believe also are subject to the approval of the Championships/Sports Management Cabinet), and there is no evidence I have been able to find of such an approval nor has the NCAA staff or Women’s Soccer Committee advised me of any such approval.  Thus I feel confident in saying that the bonus and penalty amounts ultimately used in 2010 and 2011, together with the ratings and rankings that they produced and that the Committee used in its Tournament bracket formation process, were unauthorized and incorrect.

(The NCAA staff has advised me that there are some very minor changes to the bonus and penalty amounts [in the range of 0.0001] that can be made automatically to properly calibrate the amounts in relation to expected overall ratings.  The amounts of the changes from the correct 2009 amounts to the incorrect 2010 and 2011 amounts, however, are not those kinds of changes.)

How Did the Error Occur?

The error occurred during the two week period after October 5, 2010.  During the first week, the NCAA staff person responsible for the RPI for Division 1 Women’s Soccer was conducting some experiments with the RPI.  I do not know the details of the experiments, but they apparently included testing different bonus and penalty amounts.  On completing the experiments, however, the staff person forgot to reinstall the correct bonus and penalty amounts.  This resulted in the October 12, 2010 RPI report having incorrect rankings.  At that time, knowing my rankings had matched the NCAA’s rankings for its October 5 RPI report and should have matched their rankings for the October 12 report, I advised the NCAA staff person that there was a problem with the October 12 report.  He realized he had forgotten to reinstall the correct bonus and penalty amounts and advised he would reinstall them for upcoming reports.  When the October 19 report came out, however, it again had incorrect ratings.  I again advised the NCAA staff person of this.  In response, he advised me that he had “moved the numbers back to the original origin” and that “I don’t know.”  In fact, however, as I now have shown, he did not move the numbers back to the “original origin” but instead installed the incorrect 2010 numbers.

How Did the Error Affect the End-of-Season Rankings?

I will set out below how the error affected the end-of-regular-season rankings for both 2010 and 2011.  These are the rankings the Committee used in the 2010 and 2011 Tournament bracket formation processes.  Fortunately, it appears the NCAA staff got very lucky, as it seems doubtful the error affected the at large selections in either year although it slightly altered the rankings of teams within the “bubble.”  It is not as clear as to seeding, but I’m guessing that the errors would not have affected the seeding either, at least not significantly.

The tables below show what the rankings should have been for each year for the top 60 teams, using the correct bonus and penalty amounts, compared to what they were using the incorrect amounts installed by the NCAA staff.  Where there are differences, I have noted them in bold face:

How to Avoid Similar Errors in the Future

I could have told the NCAA staff exactly what the problem was at some point during the 2010 season, if I had had access to the NCAA’s Normal RPI and  Adjusted RPI ratings on which the initial erroneous NCAA RPI reports were based (the October 12 and 19 reports).  This would have taken considerable work, but I definitely could and would have identified the exact problem before the end of the season.   Even better, I could have told the NCAA staff immediately on issuance of the October 12 and 19 reports exactly what the problem was if the NCAA also, in advance of the 2010 season, had made the bonus and penalty amounts public.

Problems like this inevitably are going to occur now and then, notwithstanding the best efforts of the NCAA staff.  Two changes will make it much more likely future errors will be caught quickly so they can be corrected before they affect the at large selection and seeding process:  (1) In the regular weekly NCAA RPI reports released publicly during the season, include the Normal RPI and Adjusted RPI ratings; and (2) In the Pre-Championship Manual for each season, include the bonus and penalty amounts that will be applicable for that season.  Whether the NCAA will be willing to do either of these things remains to be seen.