Everything is fucked: The syllabus

Published on by Sanjay Srivastava

PSY 607: Everything is Fucked
Prof. Sanjay Srivastava
Class meetings: Mondays 9:00 – 10:50 in 257 Straub
Office hours: Held on Twitter at your convenience (@hardsci)

In a much-discussed article at Slate, social psychologist Michael Inzlicht told a reporter, “Meta-analyses are fucked” (Engber, 2016). What does it mean, in science, for something to be fucked? Fucked needs to mean more than that something is complicated or must be undertaken with thought and care, as that would be trivially true of everything in science. In this class we will go a step further and say that something is fucked if it presents hard conceptual challenges to which implementable, real-world solutions for working scientists are either not available or routinely ignored in practice.

The format of this seminar is as follows: Each week we will read and discuss 1-2 papers that raise the question of whether something is fucked. Our focus will be on things that may be fucked in research methods, scientific practice, and philosophy of science. The potential fuckedness of specific theories, research topics, etc. will not be the focus of this class per se, but rather will be used to illustrate these important topics. To that end, each week a different student will be assigned to find a paper that illustrates the fuckedness (or lack thereof) of that week’s topic, and give a 15-minute presentation about whether it is indeed fucked.

Grading:

20% Attendance and participation
30% In-class presentation
50% Final exam

Week 1: Psychology is fucked

Meehl, P. E. (1990). Why summaries of research on psychological theories are often uninterpretable. Psychological Reports, 66, 195-244.

Week 2: Significance testing is fucked

Cohen, J. (1990). Things I have learned (so far). American Psychologist, 45, 1304-1312.

Rouder, J. N., Morey, R. D., Verhagen, J., Province, J. M., & Wagenmakers, E. J. (2016). Is there a free lunch in inference? Topics in Cognitive Science, 8, 520-547.

Week 3: Causal inference from experiments is fucked

Chapter 3 from: Bollen, K. A. (1989). Structural equations with latent variables. New York: Wiley.

Week 4: Mediation is fucked

Bullock, J. G., Green, D. P., & Ha, S. E. (2010). Yes, but what’s the mechanism?(don’t expect an easy answer). Journal of Personality and Social Psychology, 98, 550-558.

Week 5: Covariates are fucked

Culpepper, S. A., & Aguinis, H. (2011). Using analysis of covariance (ANCOVA) with fallible covariates. Psychological Methods, 16, 166-178.

Westfall, J., & Yarkoni, T. (2016). Statistically controlling for confounding constructs is harder than you think. PloS one, 11, e0152719.

Week 6: Replicability is fucked

Pashler, H., & Harris, C. R. (2012). Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science, 7, 531-536.

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.

Week 7: Interlude: Everything is fine, calm the fuck down

Gilbert, D. T., King, G., Pettigrew, S., & Wilson, T. D. (2016). Comment on “Estimating the reproducibility of psychological science.” Science, 251, 1037a.

Maxwell, S. E., Lau, M. Y., & Howard, G. S. (2015). Is psychology suffering from a replication crisis? What does “failure to replicate” really mean? American Psychologist, 70, 487-498.

Week 8: Scientific publishing is fucked

Fanelli, D. (2011). Negative results are disappearing from most disciplines and countries. Scientometrics, 90, 891-904.

Ioannidis, J. P. (2005). Why most published research findings are false. PLoS Med, 2, e124.

Week 9: Meta-analysis is fucked

Inzlicht, M., Gervais, W., & Berkman, E. (2015). Bias-Correction Techniques Alone Cannot Determine Whether Ego Depletion is Different from Zero: Commentary on Carter, Kofler, Forster, & McCullough, 2015. Available at SSRN: http://ssrn.com/abstract=2659409 or http://dx.doi.org/10.2139/ssrn.2659409

Van Elk, M., Matzke, D., Gronau, Q. F., Guan, M., Vandekerckhove, J., & Wagenmakers, E. J. (2015). Meta-analyses are no substitute for registered replications: A skeptical perspective on religious priming. Frontiers in Psychology, 6.

Week 10: The scientific profession is fucked

Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7, 543-554.

Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615-631.

Finals week

Wear black and bring a #2 pencil.

72 thoughts on “Everything is fucked: The syllabus”

    1. Okay as someone who runs a theology/volunteerism blog this I have to admit gave me a good chuckle Lol

      I am glad wordpress directed me to this post. It’s actually one of the most popular on all of wordpress right now! Congrats :)

  4. Hi, Sanjay.
    This post is great!
    I’d like to re-post the syllabus to a Chinese website so that more people could see it, because wordpress is blocked in China (what a shame).

    1. That’s fucked! Go for it (and if you don’t mind, please post a link here in the comments).

  8. I’m actually genuinely inspired to try to organize a journal club around this syllabus. Love it! What a great way to get people thinking critically about the scientific method!

  10. Hi Sanjay!

    I can’t believe you did this. I got notified by it from a grad student, who figured you were my twin, mainly because I have (half) joked about social psych being fucked for years…

    Is this really a real course? I mean, what the hell, it looks real enough. I am totally sucked in to this actually being a real course by your listing the location at Straub Hall, which is a real place at UOregon. And damn, it should and could be a real course. If it is … I love your colleagues. Mine gave me a mild version of Hell when I first started teaching a similar course (currently called, “The Psychology of Scientific Integrity). They were concerned that I would claim something like “Everything is fucked.” (I am not kidding…).

    I have a request, and an offer.

    Request. Down the road, would you consider adding a section (week?) titled something like, “Even when social psychology research is sound, replicable, not phacked, and uses appropriate methods and statistics, it is still fucked”? We had this paper just come out. An internal joke among the collaborators was that it should have been titled, “Is social psychology completely fucked? Well, not completely.”

    Click to access Interps%20and%20Methods,%20JESP.pdf

    In the complete absence of unreplicable findings, poor methods, or inappropriate stats, It presents case after case (after case after case) of famous, influential, “dramatic” findings that simply are not what they are cracked up to be (the New Look in perception, stereotype threat not showing “remove threat and B=W test scores,” Hastorf & Cantril not showing that Princeton and Dartmouth students saw “different” games, implicit prejudice (not) predicting discrimination, alleged but not actually delusional climate denialists, ostensible “top down” influences on perception, and more). There is plenty more where that came from, but, as this is just a comment on your blog, I will exert a massively difficult effort at self-control and stop here…

    But for an actual, real course, a grad course? You can definitely assign more than 1-2 readings/week…

    Offer. If you are actually going to be teaching this, or something like it, let me know if you want access to the required readings for my scientific integrity course.

    Lee Jussim

      1. Hi Jane. The course is my real actual course version of Sanjay’s Everything is Fucked faux course. Many readings overlap, but many do not (I was not familiar with that 1990 Meehl paper, or the Meta-Analysis is Fucked papers, and all are terrific; I will have to add them). I teach it as both an undergrad course (they get a lot of blogs and news articles, and only a handful of the scholarly articles), and as a grad course (which looks much more like “Everything is Fucked”).

        Sanjay has inspired me. I am going to post the course basics, possible at my Rutgers site, more likely as a blog on my Psych Today site. https://www.psychologytoday.com/blog/rabble-rouser. If you do not see it there in a few days, zip me an email. jussim at rci dot rutgers dot edu. The course is currently only available on Sakai, which is a really awkward and clunky teaching platform some tech-challenged U’s (like Rutgers) use.

        Hey, Sanjay, would you add my psych today site to your blog roll? I have the most fun with the “social psych is fucked” stuff, but it also touches on political biases**, what I see as a rising climate of illiberal intolerance on many campuses, how to understand demographic gaps, and education issues… The blogsite name, “rabble rouser” was my best shorthand (in 2012, when I started) for saying, “the field is fucked but no one is even listening, so I am going to sit in the back of the room and throw spit balls when the Famous and High Status Powers That Be conduct Business as Usual” and hope that someone starts actually listening. Fortunately (from my standpoint), right around that time, although criticizing business as usual remained controversial, it became downright respectable…

        ** Ok, Simine has also inspired in me an acceptance of footnotes. So, here is what I held off on with my first comment. Sanjay, please consider adding one more section:
        “Your Science and Career May Not be Totally Fucked if You Do Work on Politicized Topics and Your Data Contest Cherished Leftish Narratives Or Any Proclaimation from ‘On High,’ But You are in Some Pretty Deep Shit”

        Here are short excerpts from an as-yet unpubbed chapter, documenting individuals’ personal experiences with political biases in social psychology:

        These next two stories were provided by a former member of the Council of the American Psychological Association:
        1. “There was a vote about saying that APA thinks gay-couples are good parents. I thought the evidence was weak, especially given the idea of having a major society support that view as fact. I am all in favor of gay marriage and parenting, but did not think the evidence was at all compelling. In fact, had the evidence shown the opposite, the studies would have been shredded. I decided to abstain from the vote, but people all around me shouted at me to raise my hand so there would be a unanimous vote, and I am ashamed to say that I did so.”

        “Second vote was very similar, but on the damage done by American Indian school mascots. Again, I thought the evidence was thin– in fact, very thin in this case. But again, I voted that these mascots are harmful to students because they wanted an overwhelming vote on this issue.

        This is from a different respondent:
        “In grad school I was publicly called a ‘fucking Republican’ by a faculty member. I saw other students’ families insulted for the work they did (e.g., finance). There are other things I won’t share that were worse. Being conservative or libertarian is a stigma you can hide, so most of the comments after I learned to keep my mouth shut publicly were more climate based than directed at me. There are constant jokes about people on the right being idiots or evil. The default assumption [is that] you are liberal in any academic environment so nobody censors. I remember after one talk I went to where the speaker went off on Republicans for about 10 minutes. I had two liberal colleagues actually sympathize with me after that.”

        I have a whole section of my Rutgers site with papers on the problem of how political biases in the field distort our science:
        http://www.rci.rutgers.edu/~jussim/papers.html
        BUT those are just mine. http://heterodoxacademy.org/publications/
        and
        http://bps.stanford.edu/?page_id=2367
        have TONS of resources on this issue.

        By the way — Heterodox Academy is now open to all tenured professors. But we have a litmus test. You have to agree with this statement:
        “I believe that university life requires that people with diverse viewpoints and perspectives encounter each other in an environment where they feel free to speak up and challenge each other. I am concerned that many academic fields and universities currently lack sufficient viewpoint diversity—particularly political diversity. I will support viewpoint diversity in my academic field, my university, my department, and my classroom.”

        Sanjay and all tenured readers, consider this my personal invitation to join Heterodox.

        Lee

      2. Jane, I have now posted the actual syllabus from my actual undergrad course here:
        https://www.psychologytoday.com/blog/rabble-rouser/201608/the-psychology-scientific-integrity-syllabus/edit

        I hope to get the grad course up in a week or so. Keeping in mind that undergrads generally have only the dimmest idea of what significance test is, my actual syllabus is considerably less technical, and less sophisticated in many ways, than Sanjay’s faux syllabus; it has to be, and is, accessible to advanced undergrads.

        Lee

      4. Oh sorry. I copied the link while in worked on it and of course only I can edit. I think it will work if you leave off the “edit,” but if not you can definitely can get in by googling psych today rabble rouser.

    2. Hi Lee, sorry to disappoint but at the moment this isn’t a real course. Though I’m tempted to make it real. With Oregon’s robust academic freedom policy, I might even be able to keep the name.
      But thanks for mentioning your paper though – I saw it when it came out and I think it raises some great points. We’re on the quarter system so I limited myself to 10 weeks, but “Interpretation is fucked” certainly would slot in nicely with the theme of the class.

      1. I misread your office hours as the class hours, and found myself excited for the idea of meeting via social media over this course. I’d jump on this in a heartbeat, and plan to ask others to join in, book-club style, via facebook. Any thoughts on a possibility like this?

    3. Lee – I can’t put my finger on it exactly, but I feel like we should talk. Your post cracked me up, and you’re at Rutgers, my undergrad alma mater. My husband is more of the psychologist of the two of us, having been active in apa, but we are both interested in critical thinking and have an edited book coming out (we’re at uconn). If nothing else, I’d love to see your syllabus! I’ll look up your webpage and drop you a line.

      1. Hi Allison. See my reply to Jane above — just posted my undergrad syllabus. Be glad to talk or email. If you send something and I do not reply, there is about an 80% chance it got caught in Rutgers’ still very clunky spam filtering system… But I will look for it.

        I looked you up. I see you are working on a book on pseudoscience. If I had to choose to bet my life on findings from science or astrology, I’d take science hands down. The problem is that I still would not be all that sure I would actually live.

        Lee

      2. Allison,
        Both my grad and undergrad syllabi are now posted on my psych today site (see url’s in my replies to Jane, above).
        Lee

  13. Perfecting timing I’m in my PSY7620 Inferential Statistics learning research data statistical interpretations, my M.Ed. tells me this is some good student-centered, deep learning additions for furthering instrument verification and validation. Sweet! “)

  14. You should honestly turn this into an actual online course, where we discuss those papers (I’m in!). To the replicability crisis I would add Ramscar et. al’s discussion of why, absent of quantitatively predictive theories, completely valid findings (in social psychology) may very well not replicate across time and space, irrespective of the statistical issues and the proposed solutions that have been getting so much attention:

    Ramscar, Michael, et al. “Why many priming results don’t (and won’t) replicate: A quantitative analysis.” Manuscript, University of Tübingen (2015).

    Click to access Ramscar-Shaoul-Baayen_replication.pdf

  15. Haiku for the day:
    Everything is fucked
    In psych science research, and
    Doctor Oz is worse

  16. If you had more weeks, it’d be cool to add:
    Week 11: The solutions are fucked
    And talk about the pitfalls and possible misuse of things like preregistration, data sharing, and so forth. Because if history is any guide, there will always be a way to fuck things up. I mean, let’s not be naive optimists here, everything is fucked includes EVERYTHING :) I also want to see
    Week 12: Everything not coming out of science is mega-fucked

  18. Brilliant syllabus. I would recommend another topic:
    Is measurement in social science fucked?
    Michell, J. (1997). Quantitative science and the definition of measurement in psychology. British Journal of Psychology, 88(3), 355-383.

    1. Agreed – I was about to post Michell, J. (2008). Is psychometrics pathological science?. Measurement, 6(1-2), 7-24.

  19. I love it! My old History of Science prof (Tom Smith, univ of Oklahoma) would be all over this like a duck on a bug! I can just hear him taking all sides in a confrontational lecture. It would be great to run this logical syllabus in Computer Science!

    1. You had me at fMRi, I can’t wait until we can chat about some PET/fMRI scans, the elite around the world, since the last I read in 2015, was 4.

  25. This is great — you could cross-list it throughout the sciences, especially with some modules on graduate training, and science funding. (But shouldn’t week 7 be “everything is awesome!”?)

  27. Fuck it! Real course or not, I’m going to do the reading and course work anyway. (I’ve already got my week 10 project ready – researchers working with the Hadron just discovered
    “WELL, THAT’S STRANGE —
    Researchers orbit a muon around an atom, confirm physics is broken
    The proton’s charge radius shouldn’t change, and yet it appears to.”

    This is going to be FUN!

  29. I hope the course director does not get fucked in due course! can you list your publications? Pubpeer commenters have a look whether your research is fucked already! @rparthasarathy – we already do this in graduate courses but we don’t give glamorous title like this.

  30. Sanjay,
    for a moment you got me wishing I could go back to uni, just to sit in this course. Too bad it isn’t real, but yes, I support all those who say it should actually happen.
    Oh, you forgot the mandatory soundtrack, Dylan’s “Everything is broken” (only working & decent link I could find is this.
    Well done!

  31. dear sanjay

    i am now tempted to offer “Science is now fucked (at least in India).” at IIT Bombay (where i use “fucked” in exactly your sense).

    global rankings are fucked.

    Click to access 0461.pdf

    https://www.natureindex.com/custom-reports/indian-science-ascending/high-quality-research-output-in-a-global-context/
    admission systems are fucked.
    http://www.mydigitalfc.com/knowledge/qualify-dont-rank-975
    http://scroll.in/article/808036/part-ii-indias-healthcare-system-will-suffer-because-of-neets-bias-towards-the-cbse-syllabus
    elite institutions are fucked.
    http://www.epw.in/journal/2016/11/commentary/iits-and-project-indian-democracy.html

    Click to access RD.pdf

    big science is fucked.

    moreover, perhaps a simultaneous offering of such courses, a sort of flash-mob, may confuse administrators into promoting us into deans and chair-professors.

    regards,
    milind sohoni.

  32. Fellow escaped…ahem, “retired” colleague suggests a companion course: Everyone Is Fucked 504, to be taught in the Department of Political Science. This sets us on the way toward establishing an interdisciplinary major.

    1. I like looking it as objective reasoning … or in other words, (G Rated) darned!

  34. Great definition of modern usage in your Chronicle interview. Someone needs to update Wikipedia . . . or Webster’s. “The joke in the syllabus is “****” means either it’s unsolvable or nobody is implementing the solution.”

  35. I teach in a Malaysian university, this would be over and above the comprehension of most of the administration and the students would just memorize all reading in order to get an A

  36. This would be a great course, because we all know it gets to a deep truth. Everything really is fucked now, not just in social psych. Economics is fucked (negative interest rates?! and the central bankers of the world trying to create hyperinflation – and failing??), physics is completely fucked (Feng et al suggesting a new, fifth fundamental force? (actually might be true..), – but what does this do to Standard Model.. and what does 8Be isotope experiment say about “big science” and the LHC? (ie. you don’t need billion-dollar hadron whacker to see the anomaly).. We all know politics and government everywhere is just quad-fucked (pick your fav. example), And Business is fucked in just about every industrial and commercial domain in so many awful, ongoing, and creative ways…
    So, when i saw your course, I thought… “ah, some wise fellows from the Academy have decided to focus directly on this key fact: Never in all of human history has so much been so fucked in every area and field of inquiry…”
    As David Bowie was once told: “Don’t stop! Carry On!”
    I think you are on to something.
    It brings up the question: Just why is everything so completely and utterly fucked now?
    What do we call this new discipline? Maybe “Breakfail Theory”? (Since Fuckology sounds silly…) and what we really know is that pretty much everything is just broken and crappy stupid and awful, and in our guts or neurons, we know that it both could and should be much better.
    – Rus

  37. Just love it. Much inspiration for my own syllabi. The last final exam suggestions completely cracked me up. “Wear black and bring a no2 pencil”. It’s a bit like “Keep calm and carry on”, but much less fucked.

    1. Sorry, it would take a little while to dig up links for all of these, it may be a while before I can do that. And a fair amount of it is at closed publications. Google Scholar tends to be pretty good at finding PDFs if they’re out there somewhere. And if not, a request with the #ICanHazPDF hashtag on Twitter is another option.

    2. Hey, Stephanie,

      If you go here:
      https://www.psychologytoday.com/blog/rabble-rouser
      you will see I have two syllabi posted as blogs. Most have links to actual articles. The overlap with Sanjay’s syllabus is only partial, because these are real classes, so much of it is intro stuff, geared to students. Only about 3/4 have links to articles because I just could not find certain articles online.
      Lee

  39. I’m curious what you think of the Election Research Preacceptance Competition link. Journals will preaccept articles based only on the study questions and research design before the data are even available for analysis. It seems like a great idea to me.

    1. I think registered reports are a great idea. A few psych journals are starting to create RR submission tracks, and there’s one new journal I know of that is all registered reports (Comprehensive Results in Social Psychology). I don’t think it’s a model for how all journal articles should be done, but it’s a great option to have.

Comments are closed.