PSY 607: Everything is Fucked
Prof. Sanjay Srivastava
Class meetings: Mondays 9:00 – 10:50 in 257 Straub
Office hours: Held on Twitter at your convenience (@hardsci)
In a much-discussed article at Slate, social psychologist Michael Inzlicht told a reporter, “Meta-analyses are fucked” (Engber, 2016). What does it mean, in science, for something to be fucked? Fucked needs to mean more than that something is complicated or must be undertaken with thought and care, as that would be trivially true of everything in science. In this class we will go a step further and say that something is fucked if it presents hard conceptual challenges to which implementable, real-world solutions for working scientists are either not available or routinely ignored in practice.
The format of this seminar is as follows: Each week we will read and discuss 1-2 papers that raise the question of whether something is fucked. Our focus will be on things that may be fucked in research methods, scientific practice, and philosophy of science. The potential fuckedness of specific theories, research topics, etc. will not be the focus of this class per se, but rather will be used to illustrate these important topics. To that end, each week a different student will be assigned to find a paper that illustrates the fuckedness (or lack thereof) of that week’s topic, and give a 15-minute presentation about whether it is indeed fucked.
Grading:
20% Attendance and participation
30% In-class presentation
50% Final exam
Week 1: Psychology is fucked
Meehl, P. E. (1990). Why summaries of research on psychological theories are often uninterpretable. Psychological Reports, 66, 195-244.
Week 2: Significance testing is fucked
Cohen, J. (1990). Things I have learned (so far). American Psychologist, 45, 1304-1312.
Rouder, J. N., Morey, R. D., Verhagen, J., Province, J. M., & Wagenmakers, E. J. (2016). Is there a free lunch in inference? Topics in Cognitive Science, 8, 520-547.
Week 3: Causal inference from experiments is fucked
Chapter 3 from: Bollen, K. A. (1989). Structural equations with latent variables. New York: Wiley.
Week 4: Mediation is fucked
Bullock, J. G., Green, D. P., & Ha, S. E. (2010). Yes, but what’s the mechanism?(don’t expect an easy answer). Journal of Personality and Social Psychology, 98, 550-558.
Week 5: Covariates are fucked
Culpepper, S. A., & Aguinis, H. (2011). Using analysis of covariance (ANCOVA) with fallible covariates. Psychological Methods, 16, 166-178.
Westfall, J., & Yarkoni, T. (2016). Statistically controlling for confounding constructs is harder than you think. PloS one, 11, e0152719.
Week 6: Replicability is fucked
Pashler, H., & Harris, C. R. (2012). Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science, 7, 531-536.
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.
Week 7: Interlude: Everything is fine, calm the fuck down
Gilbert, D. T., King, G., Pettigrew, S., & Wilson, T. D. (2016). Comment on “Estimating the reproducibility of psychological science.” Science, 251, 1037a.
Maxwell, S. E., Lau, M. Y., & Howard, G. S. (2015). Is psychology suffering from a replication crisis? What does “failure to replicate” really mean? American Psychologist, 70, 487-498.
Week 8: Scientific publishing is fucked
Fanelli, D. (2011). Negative results are disappearing from most disciplines and countries. Scientometrics, 90, 891-904.
Ioannidis, J. P. (2005). Why most published research findings are false. PLoS Med, 2, e124.
Week 9: Meta-analysis is fucked
Inzlicht, M., Gervais, W., & Berkman, E. (2015). Bias-Correction Techniques Alone Cannot Determine Whether Ego Depletion is Different from Zero: Commentary on Carter, Kofler, Forster, & McCullough, 2015. Available at SSRN: http://ssrn.com/abstract=2659409 or http://dx.doi.org/10.2139/ssrn.2659409
Van Elk, M., Matzke, D., Gronau, Q. F., Guan, M., Vandekerckhove, J., & Wagenmakers, E. J. (2015). Meta-analyses are no substitute for registered replications: A skeptical perspective on religious priming. Frontiers in Psychology, 6.
Week 10: The scientific profession is fucked
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7, 543-554.
Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615-631.
Finals week
Wear black and bring a #2 pencil.
Are you going to blog weekly about the salient points?
A possible addition:
“Interdisciplinary research has consistently lower funding success”
http://www.nature.com/nature/journal/v534/n7609/full/nature18315.html
too late to change the course number to PSY 666?
Okay as someone who runs a theology/volunteerism blog this I have to admit gave me a good chuckle Lol
I am glad wordpress directed me to this post. It’s actually one of the most popular on all of wordpress right now! Congrats :)
616 (666 was mistranslation)
Hi, Sanjay.
This post is great!
I’d like to re-post the syllabus to a Chinese website so that more people could see it, because wordpress is blocked in China (what a shame).
That’s fucked! Go for it (and if you don’t mind, please post a link here in the comments).
I can read it without using VPN
Why didn’t they offer this course when I was in school?
It would be a great course and training for students
I have a PhD in EIF’d.
I’m actually genuinely inspired to try to organize a journal club around this syllabus. Love it! What a great way to get people thinking critically about the scientific method!
I agree – This is totally inspiring and I’m going to try the same at our institution!!! Thanks for sharing this!
I agree. I would both like to participate in the seminar and to offer it to students (and colleagues).
“Omnia fututam est”, ’cause everything sounds a bit smarter in latin.
Except when your Latin is f’ed too :-)
Please correct me then :) (as done in Life of Brian, if you have the time)
Hi Sanjay!
I can’t believe you did this. I got notified by it from a grad student, who figured you were my twin, mainly because I have (half) joked about social psych being fucked for years…
Is this really a real course? I mean, what the hell, it looks real enough. I am totally sucked in to this actually being a real course by your listing the location at Straub Hall, which is a real place at UOregon. And damn, it should and could be a real course. If it is … I love your colleagues. Mine gave me a mild version of Hell when I first started teaching a similar course (currently called, “The Psychology of Scientific Integrity). They were concerned that I would claim something like “Everything is fucked.” (I am not kidding…).
I have a request, and an offer.
Request. Down the road, would you consider adding a section (week?) titled something like, “Even when social psychology research is sound, replicable, not phacked, and uses appropriate methods and statistics, it is still fucked”? We had this paper just come out. An internal joke among the collaborators was that it should have been titled, “Is social psychology completely fucked? Well, not completely.”
Click to access Interps%20and%20Methods,%20JESP.pdf
In the complete absence of unreplicable findings, poor methods, or inappropriate stats, It presents case after case (after case after case) of famous, influential, “dramatic” findings that simply are not what they are cracked up to be (the New Look in perception, stereotype threat not showing “remove threat and B=W test scores,” Hastorf & Cantril not showing that Princeton and Dartmouth students saw “different” games, implicit prejudice (not) predicting discrimination, alleged but not actually delusional climate denialists, ostensible “top down” influences on perception, and more). There is plenty more where that came from, but, as this is just a comment on your blog, I will exert a massively difficult effort at self-control and stop here…
But for an actual, real course, a grad course? You can definitely assign more than 1-2 readings/week…
Offer. If you are actually going to be teaching this, or something like it, let me know if you want access to the required readings for my scientific integrity course.
Lee Jussim
Dear Lee
What is your scientific integrity course? Jane
Hi Jane. The course is my real actual course version of Sanjay’s Everything is Fucked faux course. Many readings overlap, but many do not (I was not familiar with that 1990 Meehl paper, or the Meta-Analysis is Fucked papers, and all are terrific; I will have to add them). I teach it as both an undergrad course (they get a lot of blogs and news articles, and only a handful of the scholarly articles), and as a grad course (which looks much more like “Everything is Fucked”).
Sanjay has inspired me. I am going to post the course basics, possible at my Rutgers site, more likely as a blog on my Psych Today site. https://www.psychologytoday.com/blog/rabble-rouser. If you do not see it there in a few days, zip me an email. jussim at rci dot rutgers dot edu. The course is currently only available on Sakai, which is a really awkward and clunky teaching platform some tech-challenged U’s (like Rutgers) use.
Hey, Sanjay, would you add my psych today site to your blog roll? I have the most fun with the “social psych is fucked” stuff, but it also touches on political biases**, what I see as a rising climate of illiberal intolerance on many campuses, how to understand demographic gaps, and education issues… The blogsite name, “rabble rouser” was my best shorthand (in 2012, when I started) for saying, “the field is fucked but no one is even listening, so I am going to sit in the back of the room and throw spit balls when the Famous and High Status Powers That Be conduct Business as Usual” and hope that someone starts actually listening. Fortunately (from my standpoint), right around that time, although criticizing business as usual remained controversial, it became downright respectable…
** Ok, Simine has also inspired in me an acceptance of footnotes. So, here is what I held off on with my first comment. Sanjay, please consider adding one more section:
“Your Science and Career May Not be Totally Fucked if You Do Work on Politicized Topics and Your Data Contest Cherished Leftish Narratives Or Any Proclaimation from ‘On High,’ But You are in Some Pretty Deep Shit”
Here are short excerpts from an as-yet unpubbed chapter, documenting individuals’ personal experiences with political biases in social psychology:
These next two stories were provided by a former member of the Council of the American Psychological Association:
1. “There was a vote about saying that APA thinks gay-couples are good parents. I thought the evidence was weak, especially given the idea of having a major society support that view as fact. I am all in favor of gay marriage and parenting, but did not think the evidence was at all compelling. In fact, had the evidence shown the opposite, the studies would have been shredded. I decided to abstain from the vote, but people all around me shouted at me to raise my hand so there would be a unanimous vote, and I am ashamed to say that I did so.”
“Second vote was very similar, but on the damage done by American Indian school mascots. Again, I thought the evidence was thin– in fact, very thin in this case. But again, I voted that these mascots are harmful to students because they wanted an overwhelming vote on this issue.
This is from a different respondent:
“In grad school I was publicly called a ‘fucking Republican’ by a faculty member. I saw other students’ families insulted for the work they did (e.g., finance). There are other things I won’t share that were worse. Being conservative or libertarian is a stigma you can hide, so most of the comments after I learned to keep my mouth shut publicly were more climate based than directed at me. There are constant jokes about people on the right being idiots or evil. The default assumption [is that] you are liberal in any academic environment so nobody censors. I remember after one talk I went to where the speaker went off on Republicans for about 10 minutes. I had two liberal colleagues actually sympathize with me after that.”
I have a whole section of my Rutgers site with papers on the problem of how political biases in the field distort our science:
http://www.rci.rutgers.edu/~jussim/papers.html
BUT those are just mine. http://heterodoxacademy.org/publications/
and
http://bps.stanford.edu/?page_id=2367
have TONS of resources on this issue.
By the way — Heterodox Academy is now open to all tenured professors. But we have a litmus test. You have to agree with this statement:
“I believe that university life requires that people with diverse viewpoints and perspectives encounter each other in an environment where they feel free to speak up and challenge each other. I am concerned that many academic fields and universities currently lack sufficient viewpoint diversity—particularly political diversity. I will support viewpoint diversity in my academic field, my university, my department, and my classroom.”
Sanjay and all tenured readers, consider this my personal invitation to join Heterodox.
Lee
Jane, I have now posted the actual syllabus from my actual undergrad course here:
https://www.psychologytoday.com/blog/rabble-rouser/201608/the-psychology-scientific-integrity-syllabus/edit
I hope to get the grad course up in a week or so. Keeping in mind that undergrads generally have only the dimmest idea of what significance test is, my actual syllabus is considerably less technical, and less sophisticated in many ways, than Sanjay’s faux syllabus; it has to be, and is, accessible to advanced undergrads.
Lee
Denied access to the psychtoday site :(
Oh sorry. I copied the link while in worked on it and of course only I can edit. I think it will work if you leave off the “edit,” but if not you can definitely can get in by googling psych today rabble rouser.
This link should work:
https://www.psychologytoday.com/blog/rabble-rouser/201608/the-psychology-scientific-integrity-syllabus
But you can definitely get here:
https://www.psychologytoday.com/blog/rabble-rouser
and it should pop right up.
Lee
yep, it’s there. thanks,
https://www.psychologytoday.com/blog/rabble-rouser/201608/psychology-scientific-integrity-graduate-syllabus
Grad syllabus now posted. Real course.
GREAT!!!! Thanks much.
Hi Lee, sorry to disappoint but at the moment this isn’t a real course. Though I’m tempted to make it real. With Oregon’s robust academic freedom policy, I might even be able to keep the name.
But thanks for mentioning your paper though – I saw it when it came out and I think it raises some great points. We’re on the quarter system so I limited myself to 10 weeks, but “Interpretation is fucked” certainly would slot in nicely with the theme of the class.
I misread your office hours as the class hours, and found myself excited for the idea of meeting via social media over this course. I’d jump on this in a heartbeat, and plan to ask others to join in, book-club style, via facebook. Any thoughts on a possibility like this?
I suspect this could be a wildly popular MOOC.
Another vote for this as a MOOC!
Lee – I can’t put my finger on it exactly, but I feel like we should talk. Your post cracked me up, and you’re at Rutgers, my undergrad alma mater. My husband is more of the psychologist of the two of us, having been active in apa, but we are both interested in critical thinking and have an edited book coming out (we’re at uconn). If nothing else, I’d love to see your syllabus! I’ll look up your webpage and drop you a line.
Hi Allison. See my reply to Jane above — just posted my undergrad syllabus. Be glad to talk or email. If you send something and I do not reply, there is about an 80% chance it got caught in Rutgers’ still very clunky spam filtering system… But I will look for it.
I looked you up. I see you are working on a book on pseudoscience. If I had to choose to bet my life on findings from science or astrology, I’d take science hands down. The problem is that I still would not be all that sure I would actually live.
Lee
Allison,
Both my grad and undergrad syllabi are now posted on my psych today site (see url’s in my replies to Jane, above).
Lee
Brilliant Sanjay! This certainly holds in the world of functional neuroimaging.
Is the course that proves “everything is fucked” also fucked?
Perfecting timing I’m in my PSY7620 Inferential Statistics learning research data statistical interpretations, my M.Ed. tells me this is some good student-centered, deep learning additions for furthering instrument verification and validation. Sweet! “)
You should honestly turn this into an actual online course, where we discuss those papers (I’m in!). To the replicability crisis I would add Ramscar et. al’s discussion of why, absent of quantitatively predictive theories, completely valid findings (in social psychology) may very well not replicate across time and space, irrespective of the statistical issues and the proposed solutions that have been getting so much attention:
Ramscar, Michael, et al. “Why many priming results don’t (and won’t) replicate: A quantitative analysis.” Manuscript, University of Tübingen (2015).
Click to access Ramscar-Shaoul-Baayen_replication.pdf
Haiku for the day:
Everything is fucked
In psych science research, and
Doctor Oz is worse
If you had more weeks, it’d be cool to add:
Week 11: The solutions are fucked
And talk about the pitfalls and possible misuse of things like preregistration, data sharing, and so forth. Because if history is any guide, there will always be a way to fuck things up. I mean, let’s not be naive optimists here, everything is fucked includes EVERYTHING :) I also want to see
Week 12: Everything not coming out of science is mega-fucked
Will this be on Coursera? ;)
Brilliant syllabus. I would recommend another topic:
Is measurement in social science fucked?
Michell, J. (1997). Quantitative science and the definition of measurement in psychology. British Journal of Psychology, 88(3), 355-383.
Agreed – I was about to post Michell, J. (2008). Is psychometrics pathological science?. Measurement, 6(1-2), 7-24.
I love it! My old History of Science prof (Tom Smith, univ of Oklahoma) would be all over this like a duck on a bug! I can just hear him taking all sides in a confrontational lecture. It would be great to run this logical syllabus in Computer Science!
For extra-credit, here’s a paper that shows we’re sometimes fucked even when the effects replicate reliably and are studied for 3+ decades:
http://link.springer.com/article/10.3758/s13423-014-0759-2
(Includes suggested strategies to improve the situation)
So much yes! And I’m sure several of your colleagues in the department would be happy to guest lecture for you on fMRI: what a clusterf!#^&*@k ;)
(a la Eklund et al., 2016; http://www.pnas.org/content/113/28/7900.abstract)
You had me at fMRi, I can’t wait until we can chat about some PET/fMRI scans, the elite around the world, since the last I read in 2015, was 4.
Sanjay,
If you build it, the students (and some colleagues) will come!
Gina
Gina is right!
I want to know about wearing black for the final. I was thinking purple.
This is great — you could cross-list it throughout the sciences, especially with some modules on graduate training, and science funding. (But shouldn’t week 7 be “everything is awesome!”?)
OMG. Week 7. Laughing my fucking ass off.
Ditto
Fuck it! Real course or not, I’m going to do the reading and course work anyway. (I’ve already got my week 10 project ready – researchers working with the Hadron just discovered
“WELL, THAT’S STRANGE —
Researchers orbit a muon around an atom, confirm physics is broken
The proton’s charge radius shouldn’t change, and yet it appears to.”
This is going to be FUN!
What is “real” anyway? Fuck it, I’m taking this course by myself…reading list, class projects and all. (I hope I pass…)
I’ve already got week 10 ready: http://arstechnica.com/science/2016/08/researchers-orbit-a-muon-around-an-atom-confirm-physics-is-broken/
Hadron collider physicists confirm physics is fucked. (Really)
I hope the course director does not get fucked in due course! can you list your publications? Pubpeer commenters have a look whether your research is fucked already! @rparthasarathy – we already do this in graduate courses but we don’t give glamorous title like this.
Sanjay,
for a moment you got me wishing I could go back to uni, just to sit in this course. Too bad it isn’t real, but yes, I support all those who say it should actually happen.
Oh, you forgot the mandatory soundtrack, Dylan’s “Everything is broken” (only working & decent link I could find is this.
Well done!
dear sanjay
i am now tempted to offer “Science is now fucked (at least in India).” at IIT Bombay (where i use “fucked” in exactly your sense).
global rankings are fucked.
Click to access 0461.pdf
https://www.natureindex.com/custom-reports/indian-science-ascending/high-quality-research-output-in-a-global-context/
admission systems are fucked.
http://www.mydigitalfc.com/knowledge/qualify-dont-rank-975
http://scroll.in/article/808036/part-ii-indias-healthcare-system-will-suffer-because-of-neets-bias-towards-the-cbse-syllabus
elite institutions are fucked.
http://www.epw.in/journal/2016/11/commentary/iits-and-project-indian-democracy.html
Click to access RD.pdf
big science is fucked.
moreover, perhaps a simultaneous offering of such courses, a sort of flash-mob, may confuse administrators into promoting us into deans and chair-professors.
regards,
milind sohoni.
Fellow escaped…ahem, “retired” colleague suggests a companion course: Everyone Is Fucked 504, to be taught in the Department of Political Science. This sets us on the way toward establishing an interdisciplinary major.
Please. Define “fucked” in this context.
I like looking it as objective reasoning … or in other words, (G Rated) darned!
Great definition of modern usage in your Chronicle interview. Someone needs to update Wikipedia . . . or Webster’s. “The joke in the syllabus is “****” means either it’s unsolvable or nobody is implementing the solution.”
I teach in a Malaysian university, this would be over and above the comprehension of most of the administration and the students would just memorize all reading in order to get an A
This would be a great course, because we all know it gets to a deep truth. Everything really is fucked now, not just in social psych. Economics is fucked (negative interest rates?! and the central bankers of the world trying to create hyperinflation – and failing??), physics is completely fucked (Feng et al suggesting a new, fifth fundamental force? (actually might be true..), – but what does this do to Standard Model.. and what does 8Be isotope experiment say about “big science” and the LHC? (ie. you don’t need billion-dollar hadron whacker to see the anomaly).. We all know politics and government everywhere is just quad-fucked (pick your fav. example), And Business is fucked in just about every industrial and commercial domain in so many awful, ongoing, and creative ways…
So, when i saw your course, I thought… “ah, some wise fellows from the Academy have decided to focus directly on this key fact: Never in all of human history has so much been so fucked in every area and field of inquiry…”
As David Bowie was once told: “Don’t stop! Carry On!”
I think you are on to something.
It brings up the question: Just why is everything so completely and utterly fucked now?
What do we call this new discipline? Maybe “Breakfail Theory”? (Since Fuckology sounds silly…) and what we really know is that pretty much everything is just broken and crappy stupid and awful, and in our guts or neurons, we know that it both could and should be much better.
– Rus
Just love it. Much inspiration for my own syllabi. The last final exam suggestions completely cracked me up. “Wear black and bring a no2 pencil”. It’s a bit like “Keep calm and carry on”, but much less fucked.
great stuff!! btw, can you link to live articles? my school library is underfunded and so it’s unlikely i can retrieve all these articles….
Sorry, it would take a little while to dig up links for all of these, it may be a while before I can do that. And a fair amount of it is at closed publications. Google Scholar tends to be pretty good at finding PDFs if they’re out there somewhere. And if not, a request with the #ICanHazPDF hashtag on Twitter is another option.
Hey, Stephanie,
If you go here:
https://www.psychologytoday.com/blog/rabble-rouser
you will see I have two syllabi posted as blogs. Most have links to actual articles. The overlap with Sanjay’s syllabus is only partial, because these are real classes, so much of it is intro stuff, geared to students. Only about 3/4 have links to articles because I just could not find certain articles online.
Lee
I’m curious what you think of the Election Research Preacceptance Competition link. Journals will preaccept articles based only on the study questions and research design before the data are even available for analysis. It seems like a great idea to me.
I think registered reports are a great idea. A few psych journals are starting to create RR submission tracks, and there’s one new journal I know of that is all registered reports (Comprehensive Results in Social Psychology). I don’t think it’s a model for how all journal articles should be done, but it’s a great option to have.
You should just offer this as a live online course using google hangouts or something. I’m sure since the post went viral you’d get enough people to pay to join it.
YES to Random Guy – I would pay to join it! Loved this post, just shared it at a workshop with an audience of 50, they loved it too.