From time to time I have heard people in my field challenging the usefulness of interviews in grad student selection. It usually is delivered with the weary tone of the evidence-based curmudgeon. (I should note that as an admirer of Paul Meehl and a grand-advisee of Lew Goldberg, who once wrote an article called “Human Mind Versus Regression Equation” in which the regression equation wins, I am often disposed toward such curmudeonry myself.)
The argument usually goes something like this: “All the evidence from personnel selection studies says that interviews don’t predict anything. We are wasting people’s time and money by interviewing grad students, and we are possibly making our decisions worse by substituting bad information for good.”
I have been hearing more or less that same thing for years, starting when I was grad school myself. In fact, I have heard it often enough that, not being familiar with the literature myself, I accepted what people were saying at face value. But I finally got curious about what the literature actually says, so I looked it up.
And given what I’d come to believe over the years, I was a little surprised at what I found.
A little Google Scholaring for terms like “employment interviews” and “incremental validity” led me to a bunch of meta-analyses that concluded that in fact interviews can and do provide useful information above and beyond other valid sources of information (like cognitive ability tests, work sample tests, conscientiousness, etc.). One of the most heavily cited is a 1998 Psych Bulletin paper by Schmidt and Hunter (link is a pdf; it’s also discussed in this blog post). Another was this paper by Cortina et al, which makes finer distinctions among different kinds of interviews. The meta-analyses generally seem to agree that (a) interviews correlate with job performance assessments and other criterion measures, (b) interviews aren’t as strong predictors as cognitive ability, (c) but they do provide incremental (non-overlapping) information, and (d) in those meta-analyses that make distinctions between different kinds of interviews, structured interviews are better than unstructured interviews.
If you look at point “b” above and think that maybe interviews add too little variance to be worth the trouble, my response is: live by the coefficients, die by the coefficients. You’d also have to conclude that we shouldn’t be asking applicants to write about their background or interests in a personal statement, and we shouldn’t be obtaining letters of recommendation. According to Schmidt and Hunter (Table 1), biography, interests, and references all have weaker predictive power than structured interviews. (You might want to justify those things over interviews on a cost-benefit basis, though I’d suggest that they aren’t necessarily cheap either. A personal statement plus 3 reference letters adds up to a lot of person-hours of labor.)
A bigger problem is that if you are going to take an evidence-based approach, your evidence needs to be relevant. Graduate training shares some features with conventional employment, but they are certainly not the same. So I think it is fair to question how well personnel studies can generalize to doctoral admissions. For example, one justification for interviews that I’ve commonly heard is that Ph.D. programs require a lot of close mentoring and productive collaboration. Interviews might help the prospective advisor and advisee evaluate the potential for rapport and shared interests and goals. Even if an applicant is generally well qualified to earn a Ph.D., they might not be a good fit for a particular advisor/lab/program.
That, of course, is a testable question. So if you are an evidence-based curmudgeon, you should probably want some relevant data. I was not able to find any studies that specifically addressed the importance of rapport and interest-matching as predictors of later performance in a doctoral program. (Indeed, validity studies of graduate admissions are few and far between, and the ones I could find were mostly for medical school and MBA programs, which are very different from research-oriented Ph.D. programs.) It would be worth doing such studies, but not easy.
Anyway, if I’m misunderstanding the literature or missing important studies, I hope someone will tell me in the comments. (Personnel selection is not my wheelhouse, but since this is a blog I’m happy to plow forward anyway.) However, based on what I’ve been able to find in the literature, I’m certainly not ready to conclude that admissions interviews are useless.
The Hardest Science 
The question I would want answered is whether the people you turn away because of mismatch at the interview would have performed well at your institution. I would bet (based on no evidence whatsoever) that interviews let us accept people who may not really succeed just because we liked them, but I imagine that people who are rejected because of personality mismatch would have indeed performed poorly at that institution, even if they performed well elsewhere. That’s not really testable unless you want to start accepting students you didn’t like in the interview.
I often wonder the same thing about recommendation letters, which I write for students for graduate school, and which I read when I am on search committees.
One thought I have had is that they establish a relatively low bar, but perhaps in requiring those hours of labor, the obstacle they provide is part of the selection process. In other words, part of the question that they ask is: Do you have three people who you can ask to devote an hour each to you?
Perhaps this bar prevents some people from even applying, especially those people who would not be qualified.
I will also say that I haven’t yet seen letters of recommendation count too much in search committees, but that it seems to be a bad thing when an applicant can’t seem to get any in at all.
Anyways, thanks for this.
Any quick thoughts (or links) on how to do a good structured interview for an academic position?
@Cara, that’s a pretty fundamental inference problem in evaluating any observational study of a selection procedure. How do you know how the people you didn’t select would have performed? Regarding the “likable-but-unqualified” folks, that may be the case. But any selection procedure is going to be imperfect, and thus will let some people in who it shouldn’t have. Some of them will be likable, and it will be tempting to say (in hindsight) that that’s why they got in, when in fact they may be just a byproduct of the prediction model having a residual.
@Cedar, if that’s the case, I’d rather do away with letters and just have referees spend 5 minutes making standardized ratings. By lowering the workload cost, maybe faculty would be more likely to agree to act as referees and thus they’d be less narrowly selected. (Plus you’d eliminate the dissonance-reduction problem of, “I just agreed to spend a big chunk of my precious time writing a letter for this guy, he must be great!”) As far as structured interviews, my understanding is that the first step is to make a list of job qualifications. I wonder far you’d get even on that much…
Great post! Until I read this I was also a kool-aid drinking believer in the impotence of interviews. Now I think they have at least a little bit of potence. :) That said, I think there’s still a very good reason why interviews are probably not helpful and may even be harmful, which is that people are probably not very good at weighting their personal impressions appropriately relative to the (more important) hard data. So showing that you get a better model fit if you include interviews than if you don’t doesn’t mean that faculty will make better decisions if you include interviews than if you don’t. I posted my longer thoughts here.
@Cara. By the time we get to interviews for grad school everyone invited has good letters of rec, promising statements, high GPA and GRE etc. Interviews provide additional info that help with the difficult choice of which of these people should get offers of admission. Most of the people who don’t get offers from us would probably perform well … and do, somewhere else. They wouldn’t be in the interview set if the rest of the information didn’t suggest they would. Occasionally people actually are “rejected” based on having multiple poor interviews that raised concerns about fit/suitability/match for our particular program. But mostly people are “not selected” because other people who did get offers (based on all criteria, including interviews) accepted them. If we had more openings, we *would* have selected more of the people who didn’t get offers.
I have interviewed a**holes who looked great on paper, but I would never have wanted to work with. I have also met and made offers to persons who, while they may not have had as impressive “credentials” were nevertheless impressive and knowledgeable in person, and who were possessed of intangible qualities that would not necessarily have come through on a resume or in any of the other non-interview “tools.” I like interviews–separate wheat from chaff
Hi
This is a really good discussion.
When people are talking about interviews (especially the distrust of it) I’d be interested to know what is included and excluded in this. I think there is a real continuum of face-to-face assessment (vs references, looking at publications, research aims etc) that I’ve seen in academia.
1) Dialogue to explore creativity/problem solving (I had this at my UG Psych interview)
2) Analysis of experience with research populations and my expectations of working with new ones (moving from Memory/Aging research post to Memory/Neuropysch PhD)
3) Study-specific questions on main challenges, potential challenges from reviewers, understanding of key journals (Postdoc Neuropsych/Clinical)
4) Career aims and expectations (ditto)
I assume we’re excluding presentations, but I recall colleagues having presentation-like components within their interviews. Interviews are clearly a broad church.
It wasn’t until I left academia to cross into practice (occupational/organizational psychology) that I became aware of the structured interviews that Sanjay points us to. They are quite a different breed, with features like consistent questioning and probing approaches, specific domains of exploration; for instance, behavioural event interviews and competency-based interviews only focus on past experience to uncover evidence of real behaviour rather than simply rewarding those quicker on their feet, and rating systems that link the evidence collected to pre-determined criteria.
There are pros and cons to all this – I’ve trained a lot of people that complain they don’t want to loose the freedom to ask “If you were a donut, what kind would you be?” but one person’s freedom is another’s subjectivity, and I for one am glad that selection has evolved in an evidence led-way that minimises the ‘my kinda guy’ cloning that was a risk of simply relying on free-flowing conversation.
Again as Sanjay notes, other standardised methods such as ability testing often show higher validities but a mix of methods can allow you to get to aspects that are not easy to measure psychometrically – interpersonal skills, for example.
It’s interesting to note that structured interviews can have much higher validities than much ‘sexier’ techniques such as psychometric tests of integrity – see
http://www.ncbi.nlm.nih.gov/pubmed/21319880
I might try to pick this conversation up on our new blog – a review of new evidence in the field of work psychology at http://bps-occupational-digest.blogspot.com/
Best
Alex
There is some German data for German undergraduate selection using (Germany used to have a different academic system where high school takes longer, less people go to university and everybody who goes gets a master-level degree) available.
A meta-analytic summary has been published and is available here:
Click to access Abschlussbericht_Internetversion.pdf
(Look at pages 67 for the rho values and at page 68 for—hypothetical—incremental validities)
The corrected validities even for the structured interview are quite small and the general conclusion in Germany now is that it is not worth the effort at any level.
Best, Jonas