Progress!  We are now discussing the actual structure of the ratings
systems rather than simply discussing which results fit whose agenda.
 
Dick Tuthill quotes Ken Butler's explanation of the KRACH:
 
> The above also suggests that, with few games in the database, fairly
> small changes in the data can have large effects on the rating. As a
> result, early in the season, the ratings are "jumpy", with teams
> moving apparently erratically up or down the ranking. But, as the
> season progresses, each weekend's games are a proportionally smaller
> addition to the database, so that the ratings do settle down.
 
and observes:
 
> Two points: the "jumpy" phenomenon described above is completely
> analogous to what happens with an insular schedule such as the
> MAAC's. The paucity of connecting data make this method, like the
> others, inapplicable to judging the MAAC teams against the
> establishment.
 
No matter how good the rating system, it doesn't change the fact that
we have a less-than-ideal basis for judging the relative strengths of
conferences that don't play each other.  (If the MAAC teams didn't
play the Independents, it would be impossible for *any* system to rank
them relative to anyone but each other.)  The difference between KRACH
and RPI or HEAL is that KRACH does what it can with the data
available.
 
As a "hand-waving" explanation of how the Independents act as a bridge
between MAAC and major conference teams, supposing we calculated a
KRACH rating without the Independents.  The MAAC teams would be rated
relative to one another and the established conference teams likewise.
But since multiplying the ratings by an overall constant doesn't
change things, and since the lack of games between the two groups
means we effectively have two separate rating schemes, we do not know
how the scales of the two groups match up.  We could multiply each of
them by a different constant and still obtain the correct predicted
winning percentages.  Now take the relative ratings on each side as
fixed and throw the independents into the mix, i.e., calculate the
rating each independent would need on each scale to predict the
winning percentage they have against each group of teams.  Since
Mankato and Niagara are unbeaten against the MAAC, their ratings on
the MAAC side would be at the top of the scale, while on the
established conference side, they would be somewhere in the middle
given their average performance.  Army and Air Force, on the other
hand, would be at the bottom of the major conference scale, having
gone winless against those teams, but would be somewhere in the middle
of the MAAC scale thanks to their near-.500 records.  So those four
reference points allow us to find the relative weights of the two
previously disjoint groups to each other.
 
Now, that's an oversimplification, since it doesn't consider the
effects of non-conference records on teams on each side, or the
records of independents against each other, and it doesn't really
consider the "fictitious game" used for normalization (see below).
But it is this basic effect that leads to the fact that the MAAC teams
have significantly lower KRACH ratings than most of the major
conference teams.  This also means that the KRACH is doing objectively
what most of us do when gauging the strength of the MAAC as a whole:
comparing their record vs the Division I Independents to that of the
four established conferences.
 
> Thus I do not think it can be used with any more
> reliability or credibility than any of the others unless these
> "erratic" effects are thoroughly studied.
 
It would be a good idea to play around with KRACH hypotheticals after
the season is over: how many wins over Division I independents would
it take to bring the MAAC teams up in the KRACH, how heavily would
Quinnipiac have had to dominate their conference opponents to be
ranked in the top 12, etc.
 
> The second point is that I
> recall Ken sending out an early ranking a few years ago which he may
> have modified to take out or smooth these effects.
 
Are you perhaps thinking of the "fictitious game" which each team is
assumed to have tied with a reference team whose rating is fixed to be
100, and which prevents certain pathologies while also providing a bit
of inertia to pull teams towards a 100 rating, especially when few
actual games have been played.
 
> A final question about KRACH: since the mathematics are not given,
 
Interjection: actually, the system is described in sufficient detail
in <http://www.mscs.dal.ca/~butler/krachexp.htm> that I was able to
write a program to calculate the ratings myself, and obtained a
reasonable agreement with Ken's up to an overall multiplicative factor
and the roundoff error in the posted ratings being given as integers.
 
> and
> since we know (only because Ken tells us as it is not on the top of
> our heads) that the problem is non-linear,
 
That's easy to see.  Basically, it's non-linear because A/(A+B) (the
expected winning percentage for a team with a KRACH of A vs a team
with a KRACH of B) is non-linear in A.
 
> do we know that the
> solution is single valued? Or is it possible that the error residual
> has multiple local minima which can be converged upon? And if we know
> that there is only a single solution, then how do we know that? From
> first principles, or from experience only? These are questions that
> all need to be answered before any official weight is given to another
> system in something as important as NCAA bids.
 
I don't know the answer to this (perhaps Ken or Bob does; my
background in probability theory is not extensive), but I suspect that
the introduction of "fictitious games" takes care of existance and
uniqueness of a solution to the non-linear problem.  (Without the
fictitious games, it's easy to construct situations where a team's
KRACH is undefined or undefinable, if they have only lost to
undefeated teams or beaten winless ones.)  At any rate a team's
expected winning percentage is a monotonic function of their KRACH
rating if the other teams' ratings are held constant.
 
> The "erratic" effects
> to me are the most troubling.
 
Any system is erratic when small number statistics are involved.
 
                                          John Whelan, Cornell '91
                                                  [log in to unmask]
                                     http://www.amurgsval.org/joe/
 
HOCKEY-L is for discussion of college ice hockey;  send information to
[log in to unmask], The College Hockey Information List.