Replicability in personality psychology, and the symbiosis between cumulative science and reproducible science

There is apparently an idea going around that personality psychologists are sitting on the sidelines having a moment of schadenfreude during the whole social psychology Replicability Crisis thing.

Not true.

The Association for Research in Personality conference just wrapped up in St. Louis. It was a great conference, with lots of terrific research. (Highlight: watching three of my students give kickass presentations.) And the ongoing scientific discussion about openness and reproducibility had a definite, noticeable effect on the program.

The most obvious influence was the (packed) opening session on reproducibility. First, Rich Lucas talked about the effects of JRP’s recent policy of requiring authors to explicitly talk about power and sample size decisions. The policy has had a noticeable impact on sample sizes of published papers, without major side effects like tilting toward college samples or cheap self-report measures.

Second, Simine Vazire talked about the particular challenges of addressing openness and replicability in personality psychology. A lot of the discussion in psychology has been driven by experimental psychologists, and Simine talked about what the general issues that cut across all of science look like when applied in particular to personality psychology. One cool recommendation she had (not just for personality psychologists) was to imagine that you had to include a “Most Damning Result” section in your paper, where you had to report the one result that looked worst for your hypothesis. How would that change your thinking?*

Third, David Condon talked about particular issues for early-career researchers, though really it was for anyone who wants to keep learning – he had a charming story of how he was inspired by seeing one of his big-name intellectual heroes give a major award address at a conference, then show up the next morning for an “Introduction to R” workshop. He talked a lot about tools and technology that we can use to help us do more open, reproducible science.

And finally, Dan Mroczek talked about research he has been doing with a large consortium to try to do reproducible research with existing longitudinal datasets. They have been using an integrated data analysis framework as a way of combining longitudinal datasets to test novel questions, and to look at issues like generalizability and reproducibility across existing data. Dan’s talk was a particularly good example of why we need broad participation in the replicability conversation. We all care about the same broad issues, but the particular solutions that experimental social psychologists identify aren’t going to work for everybody.

In addition to its obvious presence in the plenary session, reproducibility and openness seemed to suffuse the conference. As Rick Robins pointed out to me, there seemed to be a lot more people presenting null findings in a more open, frank way. And talk of which findings were replicated and which weren’t, people tempering conclusions from initial data, etc. was common and totally well received like it was a normal part of science. Imagine that.

One things that stuck out at me in particular was the relationship between reproducible science and cumulative science. Usually I think of the first helping the second; you need robust, reproducible findings as a foundation before you can either dig deeper into process or expand out in various ways. But in many ways, the conference reminded me that the reverse is true as well: cumulative science helps reproducibility.

When people are working on the same or related problems, using the same or related constructs and measures, etc. then it becomes much easier to do robust, reproducible science. In many ways structural models like the Big Five have helped personality psychology with that. For example, the integrated data analysis that Dan talked about requires you to have measures of the same constructs in every dataset. The Big Five provide a common coordinate system to map different trait measures onto, even if they weren’t originally conceptualized that way. Psychology needs more models like that in other domains – common coordinate systems of constructs and measures that help make sense of how different research programs fit together.

And Simine talked about (and has blogged about) the idea that we should collect fewer but better datasets, with more power and better but more labor-intensive methods. If we are open with our data, we can do something really well, and then combine or look across datasets better to take advantage of what other people do really well – but only if we are all working on the same things so that there is enough useful commonality across all those open datasets.

That means we need to move away from a career model of science where every researcher is supposed to have an effect, construct, or theory that is their own little domain that they’re king or queen of. Personality psychology used to be that way, but the Big Five has been a major counter to that, at least in the domain of traits. That kind of convergence isn’t problem-free — the model needs to evolve (Big Six, anyone?), which means that people need the freedom to work outside of it; and it can’t try to subsume things that are outside of its zone of relevance. Some people certainly won’t love it – there’s a certain satisfaction to being the World’s Leading Expert on X, even if X is some construct or process that only you and maybe your former students are studying. But that’s where other fields have gone, even going as far as expanding beyond the single-investigator lab model: Big Science is the norm in many parts of physics, genomics, and other fields. With the kinds of problems we are trying to solve in psychology – not just our reproducibility problems, but our substantive scientific ones — that may increasingly be a model for us as well.



* Actually, I don’t think she was only imagining. Simine is the incoming editor at SPPS.** Give it a try, I bet she’ll desk-accept the first paper that does it, just on principle.

** And the main reason I now have footnotes in most of my blog posts.

Norms for the Big Five Inventory and other personality measures

Every once in a while I get emails asking me about norms for the Big Five Inventory. I got one the other day, and I figured that if more than one person has asked about it, it’s probably worth a blog post.

There’s a way of thinking about norms — which I suspect is the most common way of thinking about norms — that treats them as some sort of absolute interpretive framework. The idea is that you could tell somebody, hey, if you got this score on the Agreeableness scale, it means you have this amount of agreeableness.

But I generally think that’s not the right way of thinking about it. Lew Goldberg put it this way:

One should be very wary of using canned “norms” because it isn’t obvious that one could ever find a population of which one’s present sample is a representative subset. Most “norms” are misleading, and therefore they should not be used.

That is because “norms” are always calculated in reference to some particular sample, drawn from some particular population (which BTW is pretty much never “the population of all human beings”). Norms are most emphatically NOT an absolute interpretation — they are unavoidably comparative.

So the problem arises because the usual way people talk about norms tends to bury that fact. So people say, oh, you scored at the 70th percentile. They don’t go on to say the 70th percentile of what. For published scales that give normed scores, it often turns out to mean the 70th percentile of the distribution of people who somehow made it into the scale author’s convenience sample 20 years ago.

So what should you do to help people interpret their scores? Lew’s advice is to use the sample you have at hand to construct local norms. For example, if you’re giving feedback to students in a class, tell them their percentile relative to the class.

Another approach is to use distributional information from existing dataset and just be explicit about what comparison you are making and where the data come from. For the BFI, I sometimes refer people to a large dataset of adult American Internet users that I used for a paper. Sample descriptives are in the paper, and we’ve put up a table of means and SDs broken down by age and gender for people who want to make those finer distinctions. You can then use those means and SDs to convert your raw scores into z-scores, and then calculate or look up the normal-distribution percentile. You would then say something like, “This is where you stand relative to a bunch of Internet users who took this questionnaire online.” (You don’t have to use that dataset, of course. Think about what would be an appropriate comparison group and then poke around Google Scholar looking for a paper that reports descriptive statistics for the kind of sample you want.)

Either the “local norms” approach or the “comparison sample” approach can work for many situations, though local norms may be difficult for very small samples. If the sample as a whole is unusual in some way, the local norms will remove the average “unusualness” whereas the comparison-sample approach will keep it in there, and you can decide which is the more useful comparison. (For example, an astronaut who scores in the 50th percentile of conscientiousness relative to other astronauts would be around the 93rd percentile relative to college undergrads.) But the most important thing is to avoid anything that sounds absolute. Be consistent and clear about the fact that you are making comparisons and about who you are comparing somebody to.

Two obviously wrong statements about personality and political ideology

On the heels of yesterday’s post about the link between religiosity and conservatism, I came across a New York Magazine article discussing recent research on personality, genetics, and political ideology. The article summarizes a lot of really interesting work by John Jost on ideology, Jonathan Haidt on moral foundations, David Pizarro on emotional responses and politics, etc. etc. But when it says things like…

Over the past few years, researchers haven’t just tied basic character traits to liberalism and conservatism, they’ve begun to finger specific genes they say hard-wire those ideologies.

… I just cringe. Research on personality and genetics does not support the conclusion that ideology is hard-wired, any more than our work on how political discourse ties religiosity to politics shows that ideology is a blank-slate social artifact.

Any attempt to understand the role of personality and genetics in political attitudes and ideology will have to avoid endorsing 2 obviously wrong conclusions:

1. Ideology and political attitudes have nothing to do with personality or genes.

2. Genes code for ideology and political attitudes in a clear, unconditional way.

Maybe in some distal and complex way our genes code for variations in how different psychological response systems work — under what conditions they are more and less active, how sensitive they are to various inputs, how strongly they produce their various responses, etc. In situ, these individual differences are going to interact with things like how messages are framed, how they are presented in conjunction with other information and stimuli, who is presenting the information, what we think the leaders and fellow members of our important social groups think and feel, etc.

What this interactivity means for doing science is that if you hold one thing constant (whether by experimental control or by averaging over differences) and let the other one vary, you will find an effect of the one you let vary. For example, if you look at how different people respond to the same set of sociopolitical issues, you are going to get reliable patterns of different responses that reflect people’s personalities. And if you frame and present the same issue in several different ways, and measure the average effect of the different framings, you are going to get different average responses that reflect message effects. Both are interesting experimental results, but both are testing only pieces of a plausible theoretical model.

Most researchers know this, I think. For example, from the NYMag article:

Fowler laughs at the idea that he had isolated a single gene responsible for liberalism—an idea circulated in much of the chatter about the study. “There are hundreds if not thousands of genes that are all interacting to affect complex social behaviors,” Fowler says, and scientists have only a rough sense of that process. “There’s a really long, complex causal chain at work here,” says UC-Berkeley political scientist Laura Stoker, “and we won’t get any real understanding without hundreds and hundreds of years’ more research.”

Let’s stay away from lazy and boring concepts like hard-wired. The real answers are going to be a lot more interesting.

Where does the link between religiosity and conservatism come from?

My collaborator Ari Malka has an op-ed titled Are religious Americans always conservative?

Why, then, does religiosity relate to conservatism at all? One possibility is that there is some type of organic connection between being a religious person and being a conservative person. Perhaps the traits, moral standards and ways of thinking that characterize religious people also naturally lead them to prefer conservative social outcomes and policies. Another possibility, however, is that this relation really has to do with the messages from political and religious discourse, and how some people respond to these messages.

Two pieces of evidence support this latter explanation…

The evidence comes from a new paper we have out in Political Psychology. Here’s the abstract:

Some argue that there is an organic connection between being religious and being politically conservative. We evaluate an alternative thesis that the relation between religiosity and political conservatism largely results from engagement with political discourse that indicates that these characteristics go together. In a combined sample of national survey respondents from 1996-2008, religiosity was associated with conservative positions on a wide range of attitudes and values among the highly politically engaged, but this association was generally weaker or nonexistent among those less engaged with politics. The specific political characteristics for which this pattern existed varied across ethno-religious groups. These results suggest that whether religiosity translates into political conservatism depends to an important degree on level of engagement with political discourse.

Malka, A., Lelkes, Y., Srivastava, S., Cohen, A. B., & Miller, D. T. (2012). The association of religiosity and political conservatism: The role of political engagement. Political Psychology, 33, 275-299.

The adaptive and flexible workplace personality: What should I talk about?

I was invited to give a talk at an in-service day for my university’s library staff. They are asking people from around the university to contribute, and since our library staff does so much for the rest of us, I thought it would be nice to help out. (Seriously, who doesn’t think librarians are awesome?)

The theme of the in-service day is “Exercising Your Adaptability and Flexibility” (which I think is geared toward helping people think about changes in technology and other kinds of workplace changes). The working title for my talk is “The Adaptive and Flexible Workplace Personality.” They gave me pretty wide latitude to come up with something that fits that theme, and obviously I want to keep it grounded in research. I have a few ideas, but I thought I’d see if I can use the blog to generate some more.

Personality and social psychologists, what would you talk about? What do you think would be important and useful to include? I have one hour, and the staff will be a mix of professional librarians, IT folks, other library staff, etc. I’d like to keep it lively, and maybe focus on 2 or 3 take-home points that people would find useful or thought-provoking.

Fun with Google Correlate

A new tool called Google Correlate lets you input a search term and then creates a state-by-state map of how many people search for it. It then it shows you what other search terms have similar state-by-state patterns.

A search for my name (what else would I have plugged in first?) shows the most searches coming from my home state of Oregon, and a notable lack of interest stemming from the Great Plains. Of note: interest in McBain: The Movie follows a very similar regional pattern:

Google Correlate search for sanjay srivastava and mcbain: the movie

I’m trying to think of a good scientific use for this tool, but I keep getting stuck on the fact that the top regional correlate of personality is “nipple stimulation.”

Personality traits are unrelated to health (if you only measure traits that are unrelated to health)

In the NY Times, Richard Sloan writes:

It’s true that in some respects we do have control over our health. By exercising, eating nutritious foods and not smoking, we reduce our risk of heart disease and cancer. But the belief that a fighting spirit helps us to recover from injury or illness goes beyond healthful behavior. It reflects the persistent view that personality or a way of thinking can raise or reduce the likelihood of illness.

But there’s no evidence to back up the idea that an upbeat attitude can prevent any illness or help someone recover from one more readily. On the contrary, a recently completed study of nearly 60,000 people in Finland and Sweden who were followed for almost 30 years found no significant association between personality traits and the likelihood of developing or surviving cancer. Cancer doesn’t care if we’re good or bad, virtuous or vicious, compassionate or inconsiderate. Neither does heart disease or AIDS or any other illness or injury.

Sloan, a researcher in behavioral medicine, is trying to make a point about “a fighting spirit,” but in the process he makes a larger point about personality traits being unassociated with health. And when he overreaches, he is clearly and demonstrably wrong.

That study of 60,000 people (which the Times helpfully links to) used the Eysenck Personality Inventory and thus only looked at two personality traits, extraversion and neuroticism. They found no association between those traits and incidence of cancer or survival after cancer. But the problem is that the researchers didn’t measure conscientiousness, the personality trait factor that has been most robustly associated with all kinds of health behaviors and health outcomes (including early mortality).

Of course, conscientiousness isn’t really about upbeat attitude or a fighting spirit. It’s more about diligently taking care of yourself in many small ways over a lifetime. In that respect Sloan’s central point about “fighting spirit” isn’t disputed by the conscientiousness findings. (Researchers working in the substantial optimism and health literature may or may not feel differently.) Moreover, the moral and philosophical implications — whether we should praise or blame sick people for their attitudes — go well beyond the empirical science (though they certainly can and should be informed by it). But a reader could easily get confused that Sloan is making a broader point that personality doesn’t matter in health outcomes — and that just ain’t so.

I’m not sure Sloan intended to take such a broad swipe against personality traits, given that his own research has examined links between hostility and cardiac outcomes. Then again, browsing his publications leaves me confused. His op-ed says that being “compassionate or inconsiderate” has nothing to do with heart disease; but this abstract from one of his empirical studies concludes that “[trait] hostility may be associated with risk for cardiovascular disease through its effects on interpersonal interactions.” I haven’t read his papers — I just Google Scholared him this morning — so I’ll give him the benefit of the doubt that there’s some distinction I’m missing out on.