Wednesday, March 25, 2009

Post-Mortem: It Smells Rank in Here.

We've heard it already, Wisconsin missed the NCAA tournament by a painfully small margin of 0.0002 points in the RPI. That's small. Hell, it just sounds small. Two ten-thousandths of a point. But if this is even possible, I think the margin is smaller than most realize.

Earlier in the season when I was fighting off the Valentine's Day hangover, we took a look at the PWR (ironically, focusing on the individual comparison with Ohio State) and the components therein as they stood at that time. Wisconsin was then, as it is now, the first team out. That in and of itself isn't particularly notable, but that weekend also featured one Sunday game - offering a chance to see what influence one game could have on the system.

That Sunday, Robert Morris and Bemidji State played to a 3-3 tie, and UW's RPI increased by 0.0001 points. A tie game, between two teams UW didn't play, nearly made up the entire gap that kept UW out of the tournament.

How does something like this happen?

It all stems from the RPI's formulation. When you realize that the RPI is 25% your own record, 21% the record of your opponents, and 54% the record of your opponents' opponents, it becomes more obvious how a game that's seemingly so insignificant could play such a key role.

However, that's not the way it had to be. The RPI has had numerous forumlations, ranging from the current 25-21-54 to the previous 25-50-25, and a previous version even had your winning percentage weighted at 35% of the forumla. Each iteration has problems - the 35% version meant teams with weak schedules but good records would rank too high. The 25-50-25 version often had situations where a win against a bad team would lower your RPI, or a loss against a good team could increase it. The most recent version attempted to correct this as well, but also necessitated a redefinition of the PWR's TUC criteria, which is now the top 25 teams in the RPI, whereas it was previously all teams with an RPI above .500.

Suffice it to say that the RPI isn't exactly a sacred cow. It's been reconfigured more often than Michael Jackson's face.

So what does the most recent version value? Strength of schedule? Well, not exactly. If you compare all 58 teams' RPI rank to their KRACH rank, some interesting patterns emerge. KRACH does a better job at accounting for SOS, and it does so with a more coherent methodology.

(The data above was taken from RPI, with data current through 3/21 - the most recent rankings are slightly different)

The first column is a team's rank in the RPI, and the second is that same team's rank in KRACH. The third column represents the difference between the two, color coded based on the size of the difference. The greens, for example, have an RPI that far exceeds their KRACH rating, while the reds are teams where their KRACH is better than their RPI.

Minor fluctuations of one or two spots aren't a big deal, but larger gaps indicate some systemic issues. Several WCHA schools appear under-ranked, including Minnesota, Wisconsin, and at the bottom of the rankings, Michigan Tech. Every single WCHA school has a lower RPI rating than their KRACH rating. The over-rated schools include mostly ECAC, CHA, and AHA teams.

Take Air Force, for example. Their SOS compoents rank fairly low in both the RPI and KRACH - 45th and 51st, respectively. Yet the RPI gives Air Force the 17th best rating in the country, while KRACH puts them in 29th place.

Interpret this information how you will, but it seems clear that the current iteration of the RPI (and the PWR that relies on it so heavily) undercounts strength of schedule, thus systematically putting teams from stronger conferences at a disadvantage.