Self-selection into online or face-to-face studies

A new paper by Edward Witt, Brent Donellan, and Matthew Orlando looks at self-selection biases in subject pools:

Just over 500 Michigan State University undergrads (75 per cent were female) had the option, at a time of their choosing during the Spring 2010 semester, to volunteer either for an on-line personality study, or a face-to-face version…

Just 30 per cent of the sample opted for the face-to-face version. Predictably enough, these folk tended to score more highly on extraversion. The effect size was small (d=-.26) but statistically significant. Regards more specific personality traits, the students who chose the face-to-face version were also more altruistic and less cautious.

What about choice of semester week? As you might expect, it was the more conscientious students who opted for dates earlier in the semester (r=.-.20). What’s more, men were far more likely to volunteer later in the semester, even after controlling for average personality difference between the sexes. For example, 18 per cent of week one participants were male compared with 52 per cent in the final, 13th week.

Self-selection in subject pools is not a new topic — I’ve heard plenty of people talk about an early-participant conscientiousness effect (though I don’t know if that’s been documented or if it’s just lab-lore). But the analyses of personality differences in who takes online versus in-person studies are new, as far as I know — and they definitely add a new wrinkle.

My lab’s experience has been that we get a lot more students responding to postings for online studies than face-to-face, but it seems like we sometimes get better data from the face-to-face studies. Personality measures don’t seem to be much different in quality (in terms of reliabilities, factor structures, etc.), but with experiments where we need subjects’ focused attention for some task, the data are a lot less noisy when they come from the lab. That could be part of the selection effect (altruistic students might be “better” subjects to help the researchers), though I bet a lot of it has to do with old-fashioned experimental control of the testing environment.

What could be done? When I was an undergrad taking intro to psych, each student was given a list of studies to participate in. All you knew was the codenames of the studies and some contact information, and it was your responsibility to arrange with the experimenter to take the experiment. It was a pain on all sides, but it was a good way to avoid these kinds of self-selection biases.

Of course, some people would argue that the use of undergraduate subject pools itself is a bigger problem. But given that they aren’t going away, this is definitely something to pay attention to.

McAdams on Bush: a psychobiography

Personality psychologist Dan McAdams has a new book out called George W. Bush and the Redemptive Dream. Dan was my undergraduate advisor, and I saw him give a provocative talk about this work at last summer’s ARP conference. I just told my wife to add the book to my Christmas list.

Most of McAdams’s research centers on personal narratives — the stories that people create and tell about themselves, and what role these stories play in identity and personality. But in the talk — and I gather in the book as well — Dan drew on a variety of theories and frameworks to understand some of Bush’s most consequential actions before and during his time in office. Here’s a brief description from an announcement I got about the book:

This short, streamlined psychological biography uses some of the best scientific concepts in personality and social psychology to shed light on Bush’s life, with a focus on understanding his fateful decision, as President, to launch a military invasion of Iraq.  The analysis draws heavily from contemporary research on Big Five traits, psychological goals and strivings, and narrative identity, as well as social identity theory, evolutionary psychology, research on motivated social cognition, research on authoritarianism and related concepts in political psychology, and Jon Haidt’s brilliant synthesis of moral intuitions.

Once upon a time, psychobiography was a pretty well-respected enterprise in personality psychology. I think it’s fallen out of favor in part because of the field’s emphasis on the Big Five traits and other discrete, fractionated variables. That emphasis has had benefits, focusing the field on constructs and theories that we can rigorously quantify and formalize.

But early personality psychologists like Gordon Allport and Henry Murray emphasized that any comprehensive study of personality must be able to account for the person as an integrated whole and a unique individual. The field has lost track of that to a substantial degree. But unlike earlier psychobiographers, who had very little and/or bad science to draw upon, McAdams has almost a century worth of theories and empirical research to bring to bear. That doesn’t mean the task is easy now. But I’m definitely looking forward to reading how Dan took it on.

Is there anything special about the Five-Factor Model?

I recently put up a clip-job list of all the ideas I’ve been too busy or lazy to flesh out into real posts in the last month. One of the items was about a recent Psych Inquiry commentary I wrote in response to a piece by Jack Block. Tal actually read the commentary (thanks, Tal!) and commented:

…What I couldn’t really get a sense of from your paper is whether you actually believe there’s anything special about the FFM as distinct from any number of other models, or if you view it as just a matter of convenience that the FFM happens to be the most widely adopted model. I suspect Block would have said that even if you think the FFM is all in the eyes of the beholder, there’s still no good reason to think that it’s the right structure, and that with only slightly different assumptions and a slightly different historical trajectory, we could all have been working with a six or seven-factor model. So I guess my question would be: should one read the title of your paper as saying that the FFM is the model that describes the structure of social perceptions, or are you making a more general point about all psychometric models based on semantically-mediated observations?

That’s a great question.

As I think I make clear in the paper, I think it’s highly unlikely that the FFM is isomorphic with some underlying, extra-perceptual reality of bodies or behavior. In other words, I don’t expect we’ll find five brain systems whose functioning maps one-to-one onto the five factors. I could be wrong, but I have seen exactly zero evidence that makes me think that’s the case.

But since I argue in the paper that the FFM is a model of the social concerns of ordinary social perceivers, I think it’s fair to ask whether it’s isomorphic with something else. Like maybe there are five basic, universal social concerns that all humans share, or something like that. And my answer is… no, I don’t think so.

For one thing, I don’t think the cross-cultural evidence is strong enough to support that conclusion. (Being in the same department as Gerard Saucier has helped me see that.) McCrae and Costa have done a very good job of showing that the FFM can be exported to other cultures — if we give people the FFM as a meaning system, they’ll use it in roughly the way we expect. But emic studies have been a lot more varied.

I also am not convinced that factor analysis — a method that derives independent factors from between-person covariance structures — is the “true” way to model person perception and social meaning. Useful? As a way of deriving a descriptive/taxonomic model, absolutely. Orthogonal factor analysis has some very useful properties, like mapping a multidimensional space very efficiently. And there’s a consistent something behind that useful model, in the sense that something is causing that five-factor structure to replicate (conditional on the item selection procedures, samples from certain cultures, statistical assumptions, etc.).

But there’s no reason to think that that means the five-factor structure has a simple, one-to-one relationship to whatever reality it’s grounded in — whether the reality of target persons’ behavior or of perceivers’ concerns. Why would social concerns be orthogonal (and by implication, causally unrelated to one another)? Why, if these are major themes in human social concerns, don’t we have good words for them at the five-factor level of abstraction? (“Agreeableness”? Blech. Worst factor label ever.) Why do they emerge in the between-person covariance structure but not in experimental methods that probe social representation at the individual level (ala Dabady, Bell, & Kihlstrom, 1999)?

As to Tal’s last question (“are you making a more general point about all psychometric models based on semantically-mediated observations?”): I think I say this in the paper, but I don’t think there is, or ever will be, any structural model of personality that isn’t pivotally dependent on human perception and judgment. (Ouch, double negative. Put more straightforwardly: all models of personality depend on human interpretations of personality.) I have a footnote where I comment that the Q sort can be seen as a model of what Jack Block wants to know about persons. I’ll even extend that to models that use biological constructs as their units rather than linguistic ones, but maybe I’ll save that argument for another day…

Apparently I’m on a blogging break

I just noticed that I haven’t posted in over a month. Don’t fear, loyal readers (am I being presumptuous with that plural? hi Mom!). I haven’t abandoned the blog, apparently I’ve just been too busy or preoccupied to flesh out any coherent thoughts.

So instead, here are some things that, over the last month, I’ve thought about posting but haven’t summoned up the wherewithal to turn into anything long enough to be interesting:

  • Should psychology graduate students routinely learn R in addition to, or perhaps instead of, other statistics software? (I used to think SPSS or SAS was capable enough for the modal grad student and R was too much of a pain in the ass, but I’m starting to come around. Plus R is cheaper, which is generally good for graduate students.)
  • What should we do about gee-whiz science journalism covering social neuroscience that essentially reduces to, “Wow, can you believe that X happens in the brain?” (Still working on that one. Maybe it’s too deeply ingrained to do anything.)
  • Reasons why you should read my new commentary in Psychological Inquiry. (Though really, if it takes a blog post to explain why an article is worth reading, maybe the article isn’t worth reading. I suggest you read it and tell me.)
  • A call for proposals for what controversial, dangerous, or weird research I should conduct now that I just got tenure.
  • Is your university as sketchy as my university? (Okay, my university probably isn’t really all that sketchy. And based on the previous item, you know I’m not just saying that to cover my butt.)
  • My complicated reactions to the very thought-provoking Bullock et al. “mediation is hard” paper in JPSP.

Our spring term is almost over, so maybe I’ll get to one of these sometime soon.

New resource for interpersonal perception researchers

Via Dave Kenny, I just found out about a new set of resources for researchers interested in personality and social relationships — and especially for users of the Social Relations Model.

Persoc is a research network founded by a group of mostly German researchers, although they seem to be interested in bringing people together from all over. From their website:

In September 2007 a group of young researchers who repeatedly met at conferences realized that they were all fascinated by the complex interplay of personality and social relationships. While we studied the effects of personality on very different social processes (e.g., zero acquaintance judgments, group formation, friendship development, mate choice, relationship maintenance), we shared a strong focus on observing real-life phenomena and implementing advanced methods to analyze our data. Since the official start of Persoc in late 2008, several meetings and workshops have deepened both, our interconnectedness as well as our understanding and interest in personality and social relationships. Persoc is funded by the German Research Foundation (DFG).

Among other things, they have created an R package called TripleR for analyzing round-robin data using the SRM componential approach. TripleR is intended as an alternative to the venerable SOREMO software created by Kenny. The persoc website also includes a page discussing theoretical concepts in interpersonal perception, an overview of a number of useful research designs, and other information.

Rhymes with schmersonality

Kirstin Appelt of the Center for Research on Environmental Decisions at Columbia has put together a nifty online index of personality measures. It’s called the Decision Making Individual Differences Inventory, which abbreviates to DMIDI. The email announcement I just got helpfully points out that the name “rhymes with ‘p. diddy’.” That may be my second-favorite part.

My first favorite part is that it’s a cleanly-designed, well-put-together website that looks like it will have tons of useful information for researchers. And even though it has an emphasis on measures relevant to decision-making research, the site casts a pretty wide net, including a number of Big Five measures and measures of “cognitive ability.” The latter gets snoot-quotes because for some reason economists and JDM researchers don’t like the word “intelligence.” (For that matter, they have a pretty narrow view of the word “personality” too. The section for trait measures is simply labeled “personality,” which is somehow placed in contrast to measures of motivation, attitudes, cognitive style, and ability — all of which, of course, are part of what makes you the person that you are, i.e., your personality.)

But I digress. It’s still under construction, but it looks like it will be a great resource. The site is set up as a wiki, which raises the possibility that they’ll be able to harness the academic community’s energy in updating and expanding it. I can see why they might be cautious about going down that road (who wants to moderate an edit war between a bunch of cantankerous professors?), but even in its current form it’s really nice.

Personality, economics, and human development

Just back from the Association for Research in Personality 2009 conference in Evanston. Lots of interesting stuff.

One of the main themes underlying the conference was integration with economics. There were (nominally) 2 symposia on personality and economics, as well as a keynote from James Heckman.

I say “nominally” because one of the symposia was really just a bunch of psychologists using an economics panel study (the SOEP) to study personality and life satisfaction. Very interesting stuff — the size of the dataset allows them to use some very sophisticated quantitative models (though I had some quibbles with them not including systematic growth functions) — but it didn’t feel to me like it was very far outside of the mainstream personality psychology paradigm.

One of the highlights for me, though, was Heckman’s keynote address.

First, what it wasn’t: when I first heard that a big-shot economist was getting interested in personality, I assumed he wanted to use personality traits to predict economically relevant behaviors, like how people form preferences and deal with uncertainty. It sounded like a good idea, because many economists (and their psychologist cousins in decision-making) have traditionally been strong situationists and thus resistant to thinking that personality matters. And in fact, that’s what one of the talks in the actually-about-economics symposium was about (as well as some emerging work elsewhere in DM) — how personality predicts economic decisions. It’s good and important stuff, if maybe a little unsurprising as a general direction to go.

But Heckman is interested in personality in a different way. In particular, he is interested in personality development and change. His interest grows out of research showing that interventions designed to lift people (esp. young kids) out of poverty (like the Perry Preschool Study, a precursor to Head Start) are working — kids who receive early care and educational help are more likely to go on to graduate from high school, more likely to be employed full-time as adults, less likely to get involved in crime, etc. Where Heckman got involved is in understanding the mechanisms. His work has shown that these programs don’t just boost cognitive skills (that’s economist-speak for IQ) — in fact, gains in tested IQ fade a few years after the intervention. Instead, the interventions seem to be mediated by lasting changes what economists call “noncognitive skills,” which is a slightly hilarious (if you’re a psychologist) term for personality. Enduring changes in things like diligence, cooperation, positive social relationships, etc. are what seem to be driving the effects. In Big Five terms, agreeableness and conscientiousness.

Not only is it refreshing to see an economist getting interested in personality (and as a sidenote, with what I took as a very authentic interest in making it a true 2-way street), but it’s refreshing to see anybody view personality as something that is subject to change via environmental inputs. That’s a drum I’ve been banging for a while, and the field is starting to come back to that as an interest (not only or even substantially because of my drum-banging — people like Brent Roberts, Ravenna Helson, Rebecca Shiner, Dan Mroczek, Avshalom Caspi, etc. have been banging it way longer than I have). But the Q&A showed that there’ll be some resistance. One of the presenters from the life-satisfaction panel — in fact, the one who seemed somewhat resistant to including systematic growth in his models — tried to challenge Heckman on that point, suggesting (wrongly in my view) that traits are too stable to be meaningful targets for intervention.

The same questioner also raised what I thought was a more interesting point, which is, isn’t a bit creepy to be thinking about public-policy interventions designed to mold personality? Heckman’s answer was a good start though maybe a little unsatisfying. He basically said that he sees what he’s doing as empowering people to act on their preferences. (Hence the economists’ “skills” rather than “personality.”) If you’re more capable of being cooperative and diligent, you can still choose a life of poverty and crime if you want it, but you are now empowered with the wherewithal to obtain and keep a decent job if that’s what you would really prefer. This harkens back to Wallace’s (1966) abilities conception of personality, which maybe could stand for a dusting-off.