The Chronicle blog reports that the NRC just released the methodology for its long awaited program quality rankings. The actual rankings are expected sometime this year.
NRC rankings are sort of like US News rankings, except (a) they’re specifically about doctoral programs and thus more heavily research-focused, and (b) faculty and administrators don’t feel quite as obliged to pretend they ignore the NRC rankings the same way they pretend to ignore US News. The method that the NRC came up with is pretty complex — but there’s a decent flowchart overview in the methodology handbook.
The core problem for the NRC is deciding how to combine all the various program characteristics they collect — stuff like numbers of publications, grants, citation rates, etc. — into a single dimension of quality. So they decided to come at it a couple of ways. First, they surveyed faculty about how much various attributes matter. (Not a direct quote, but along the lines of, “How important are grants in determining the overall quality of a program?”) Second, they asked faculty respondents to rank a handful of actual programs, and then they used regressions to generate implicit weights (so e.g. if the programs that everybody says are the best are all big grant-getters, then grants get weighted heavily). The explicit and implicit weights were then combined. Everything was done separately field-by-field.
What’s both cool and crazy is that they decided to preserve the uncertainty in the weights. (e.g., some respondents might have said that grants are the most important thing, others said grants are less important.) So they are going to iteratively resample from the distribution of weights, and for each program they will produce and report a range of rankings instead of a single ranking. (Specifically, it looks like they’re going to report the interquartile range.) So for each program, they’ll report something like, “Among all basketweaving departments, yours is ranked somewhere between #7 and #23.”
This immediately makes me think of 2 things:
1. Will they make the primary data available? As a psychologist, I’d think you could have a field day testing for self-serving biases and other interesting stuff in the importance ratings. There’s all kinds of interesting stuff you could do. For example, if an individual doesn’t get a lot of grants but is in a department that rakes it in, would they show a “department-serving bias” by saying that grants are important, or a true self-serving bias by saying that they aren’t? Would these biases vary by field?
2. When the actual numbers come out, will top programs disregard the ranges and just say that they’re number 1? If the upper bound of your range is #1 and your lower bound is better than everybody else’s lower bound, you’ve got a reasonable case to say you’re the best. I have a feeling that programs in that position will do exactly that. And the next-highest program will say, “We’re indistinguishable from Program A, so we’re #1 too.”
The Hardest Science 