Just to say a couple of things before I launch in:
 
(a) if you think "the KRACH guy" has been quiet amid all this discussion,
that's because I was subscribed only to info-hockey-l until a couple of
days ago, a state of affairs I have now put right.
(b) there's a lot of statistical stuff in here. Feel free to move swiftly
to the next posting if that is not your thing.
 
Bob Stagat wrote (in response to Dick Tuthill in response to KRACH):
>
> Actually, with some (considerable?) more effort, the KRACH could estimate
> not only the mean value of each team's rating, but also the relative
> uncertainty in those estimates -- a so-called "likelihood interval." Actually,
> one could do it for an individual team without too much difficulty if one
> assumed all its opponents ratings were fixed at their estimated values,
> then treat that individual team's rating as a variable.
 
Having spent much of the evening thinking about this, I'd come down on the
side of "considerable".
 
It is doable, though. Bob's approach will give a so-called "profile
likelihood interval", which has a solid statistical pedigree. For a couple
of reasons, though, I've gone with something a bit different (equally
valid, however).
 
After thinking about the issues for a bit, I felt that the real issue here
is not so much the uncertainty in single teams' ratings but the
uncertainty in comparisons between pairs of teams -- "can we conclude that
these two teams are significantly different in strength?". For any
particular pair of teams, this depends not only on how much we know about
the teams' strengths, but also on how "well-connected" they are. For
instance, two teams in the same conference are well connected because not
only will they have played each other, but also they have several
opponents in common. We will typically know quite a lot about the relative
strengths of teams like these, and much less about the relative strengths
of two teams in different conferences.
 
There's a more-or-less standard way of getting estimates of variability
out of a statistical model; it uses an array called the "information
matrix", which is here 52 x 52 because of the 52 teams. The inverse of
this matrix gives the (estimated) variances of the ratings and the
covariances between them (which reflect facts such as that a particular
game affects the ratings of its participants the most, their conference
rivals a little, and the other teams hardly at all -- the
well-connectedness thing again). From this information matrix you can also
figure out the uncertainty in the difference between two teams' ratings.
(On the scale of my calculations, the difference is the right thing to
look at; on the scale of the published KRACH values, they should be
compared by dividing them.) Then I can take the difference in ratings (on
the appropriate scale) and divide by the uncertainty (standard error of
the difference) to get a z-statistic -- and by the theory, it is
(approximately) correct to compare this with a normal distribution to
decide whether the two teams I'm comparing differ significantly in
strength.
 
I did this for all pairs of the 52 teams, and got the table below.  I
should explain the layout: down the left side are the teams, in KRACH
order, with their ratings; across the top are the same teams in the same
order, represented by their initial letter. In the table itself, each pair
of teams is represented by a number, with the following meaning:
 
 
  =: Not significantly different at the 10% level
  0: Significantly different at the 10% level
  5: Significantly different at the 5% level
  1: Significantly different at the 1% level
  *: Significantly different at the 0.1% level
 
Team       Rating NNMMCBCMDSNNORPCPMFBBMYSHMCWVNMMNLABMADWQMMCNHAUACIF
N Dakota      861 x======05555555111111*1*1*1*************1***********
New Hampshire 807 =x=====00555555111111111*1**************1***********
Maine         677 ==x======00005551111111111111*1*1*******1***********
Michigan St   514 ===x======000==005151111111111*11*111*1*1**1********
Colorado Coll 377 ====x=============00055151515511111111115**1********
Boston Coll   352 =====x===========005005555551111111111115**1********
Clarkson      327 ======x============0005055555555111111115**1*1******
Michigan      258 00=====x===================00050015551510111*1******
Denver        258 50======x==============0=0=00000051555510**1*1******
St Lawrence   239 550======x================0=0000000555150111*1******
N Michigan    219 5500======x===================0=050001510115*1******
Notre Dame    208 5500=======x==================0==5000501=115*11*****
Ohio State    202 5500========x=================0==0000501=115*11*****
Rensselaer    201 555==========x=====================00055=115111*****
Princeton     201 555===========x====================00050=115111*****
Colgate       181 1150===========x===================00000=115111**1**
Providence    175 1110============x===================0000=115111**1**
Mass Lowell   143 1115=0===========x=======================5501511*11*
Ferris State  132 111100============x======================55=1511111*
Boston U      132 1115050============x=====================55=1511111*
Bowling Green 125 1111000=============x====================55=1511111*
Minnesota     124 *111500==============x===================51=1511111*
Yale          113 1111555===============x==================00=5011111*
St Cloud      110 *111150=0==============x=================55=1011111*
Harvard       106 1*11555=================x================00=5051111*
MSU-Mankato   104 *111155=0================x===============00=1511111*
Cornell       103 1*11555==0================x===============0=5051111*
Wisconsin      98 **1115500==================x=============00=5051111*
Vermont        96 **11515000==================x=============0=5051111*
Northeastern   92 ***1515000===================x==============0051111*
Miami          91 **1*115500000=================x=============0051111*
Mass Amherst   90 ***1115000=====================x============0051111*
Niagara        82 **111110000=====================x===========0=55111*
Lake Superior  80 ****111150550====================x==========0=05151*
AK-Anchorage   78 ***1111510000=====================x=========0=05151*
Brown          76 ***1111555000000===================x==========05151*
Merrimack      72 ***11115550000000===================x=========051511
AK-Fairbanks   66 ****1111551550000====================x=========05551
Dartmouth      65 ***11115515005500=====================x========55551
W Michigan     61 ****1111151115000======================x=======05051
Quinnipiac     53 11115550000=============================x=======0511
Michigan Tech  39 *******1*11111111555550500=0=============x========01
MN-Duluth      36 *******1*11111111555510500000=============x=======01
Connecticut    35 ***111111155555550=========================x=====051
Nebraska-Omaha 29 *************1111111115151555000000=========x======1
Holy Cross     23 ******11111111111555550005000000=============x=====1
Air Force      23 ***********11111111111115155555550000=========x====1
Union          18 *****************11111111111111155555050=======x===1
Army           13 ******************11111111111111111115550=======x==5
Canisius       11 ***************11111111111111111155555505==0=====x=5
Iona            8 *****************111111111111111111115551005======x5
Fairfield       0 ************************************111111111111555x
 
 
The general story is that there is an *awful* lot of uncertainty about
how the teams stack up against each other. Once you get down to the
"bubble" teams, there's a range of 20 or more places where teams are not
significantly different in strength, according to KRACH at least. For
example, #12 Notre Dame is not significantly weaker than #5
Colorado College, and yet neither significantly stronger than
#34 Niagara (as judged from the long string of = signs). You can argue
whether that's a failing of KRACH or a failing of the schedule; I would
claim that the schedule is at least partly to blame, in that we know very
little (by any criterion) about how two teams in different conferences
compare with each other.
 
Hands up all those of you who searched for Quinnipiac? :-)
 
> I'd try messing
> with that except, because USCHO is down I'm not able access anybody's
> complete record for the season. When USCHO recovers I might try playing with
> that. My guess is that the uncertainties in the MAAC teams' ratings are
> probably too small to consider promoting any of them from the bottom 12 to the
> top 12. But at the moment that's sheer speculation on my part.
 
Well, you have your answer, kind of. Quinnipiac has a huge range of
uncertainty: all that can be said is that they are significantly weaker
than the top 11 and stronger than the bottom 4, which is not much return
from 24 games' worth of information. The trouble is, very few of those
games actually carried much information; most of them were wins against
MAAC rivals that could almost have been phoned in without the teams taking
the ice.
 
Connecticut, another MAAC team, also has a longer range of
not-significantly-differents than the nearby teams in the list. On the
other hand, it is comforting to be able to prove Fairfield significantly
weaker than absolutely everybody else...
 
Cheers,
Ken.
 
--
Ken Butler  /  [log in to unmask]  /  http://mscs.dal.ca/~butler
                 Tants caps, tants barrets.