Almost a decade ago, a scientific misconduct scandal shocked social psychologists. A prominent researcher on a career fast track was discovered to have fabricated data and committed other forms of misconduct for 4 articles in prominent journals (JPSP, PSPB, and Psychological Science). The studies were funded by NIH and published while she was at Harvard. A joint investigation by NIH and Harvard resulted in the researcher, Karen Ruggiero, admitting to misconduct, retracting the articles, and leaving academia.

The other day I read a post by Ben Goldacre about a new blog called Retraction Watch that follows scientific retractions. Goldacre mentions a study that followed up on citations of a retracted article from the late ’80s. That immediately reminded me of the Ruggiero incident, which was a big deal when I was a grad student. And it made me wonder: are people still citing Karen Ruggiero’s retracted papers?

Fortunately, it’s relatively easy to do a quick check via Google Scholar — just look up the (now-retracted) articles, click the “Cited by” link, and count the number of hits. The investigation report came out in December 2001, and the last of the retractions was published in March 2002. We should probably allow for some publication lag, so let’s forgive anything with a publication year of 2002 or earlier. How many citations are there from 2003 onward?

Here’s what I found:

• Ruggiero, K.M. & Marx, D.M (1999). Less pain and more to gain: Why high-status group members blame their failure on discrimination. Journal of Personality and Social Psychology, 77, 774-784. Cited 7 times since 2003. (Google Scholar gives 9 hits, but 2 appear to be duplicates.)
• Ruggiero, K.M., Steele, J., Hwang, A., & Marx, D.M. (2000). Why did I get a ‘D’?  The effects of social comparisons on women’s attributions to discrimination. Personality and Social Psychology Bulletin, 26, 1271-1283. Cited 2 times since 2003.
• Ruggiero, K.M. & Major, B.N. (1998). Group status and attributions to discrimination:  Are low- or high-status group members more likely to blame their failure on discrimination? Personality and Social Psychology Bulletin, 24, 821-838. Cited 9 times since 2003.
• Ruggiero, K.M., Mitchell, J.P., Krieger, N., Marx, D.M., & Lorenzo, M.L. (2000). Now you see it, now you don’t:  Explicit versus implicit measures of the personal/group discrimination discrepancy. Psychological Science, 22, 57-67. Cited 3 times since 2003.

[Let me pause here to note that the investigation concluded Ruggiero acted alone. I have listed complete citations, but please keep in mind that her co-authors were not responsible for the misconduct.]

Are these numbers a lot or a little? You can judge for yourself, but it’s at least worth noting that all are greater than zero. Some of the citations are as recent as 2010. I did not read the citing articles, but based on the titles, none appeared to be discussions of scientific misconduct; all seemed to be related to the substance of the retracted papers.

How could that happen? Since most people find articles to cite through electronic databases, I thought I’d take a look to see how these articles are listed. When I looked up these articles in PsycINFO, all 4 listings clearly state that the articles were retracted. But that isn’t true of other databases.

When I looked up the JPSP article in Google Scholar, clicking on the title took me to a ScienceDirect link (ScienceDirect is a product of Elsevier, although Elsevier does not publish JPSP). The ScienceDirect listing contained the title, abstract, etc. but did not say anything about the article having been retracted.

Clicking on both of the PSPB articles and the Psychological Science article in Google Scholar led me to entries in the Sage Journals Online database (Sage publishes those journals). None of those links mentioned the retractions. In fact, in addition to the usual information on each article (title, abstract, etc.), Sage goes a step further and lists other articles that cite them!

Google Scholar seems to give different links depending on whether you are on an institutional network that has access to certain databases, so your results may vary depending on where you are (and perhaps what search terms you use). However, it’s also worth noting that Google Scholar itself did not flag the articles as having been retracted. Sometimes my searches separately brought up the retraction notices on the same page (but always lower), and sometimes they didn’t.

Perhaps worst of all, when I retrieved the electronic full text of all four articles, I got the articles in their original form. None of the articles was marked to indicate that the article had been subsequently retracted.

Is this a problem? I think it is. Papers do get corrected or retracted from time to time (not always for sinister reasons like scientific misconduct), and it is important that researchers don’t keep citing them. I don’t know if this is an anomaly, but it does make you wonder if some databases are not being properly updated with retractions — and what effect that is having on science.