Improving the grant system ain’t so easy

Today’s NY Times has an article by Gina Kolata about how the National Cancer Institute plays it safe with grant funding. The main point of the article is that NCI funds too many “safe” studies — studies that promise a high probability of making a modest, incremental discovery. This is done at the expense of more speculative and exploratory studies that take bigger risks but could lead to greater leaps in knowledge.

The article, and by and large the commenters on it, seem to assume that things would be better if the NCI funded more high-risk research. Missing is any analysis of what might be the downsides of adopting such a strategy.

By definition, a high-risk proposal has a lower probabilty of producing usable results. (That’s what people mean by “risk” in this context.) So for every big breakthrough, you’d be funding a larger number of dead ends. That raises three problems: a substantive policy problem, a practical problem, and a political problem.

1. The substantive problem is in knowing what would be the net effect of changing the system. If you change the system so that you invest grant dollars in research that pays off half as often, but when it does the findings are twice as valuable, it’s a wash — you haven’t made things better or worse overall. So it’s a problem of adjusting the system to optimize the risk X reward payoffs. I’m not saying the current situation is optimal; but nobody is presenting any serious analysis of whether an alternative investment strategy would be better.

2. The practical problem is that we would have to find some way to choose among high-risk studies. The problem everybody is pointing to is that in the current system, scientists have to present preliminary studies, stick to incremental variations on well-established paradigms, reassure grant panels that their proposal is going to pay off, etc. Suppose we move away from that… how would you choose amongst all the riskier proposals?

People like to point to historical breakthroughs that never would have been funded by a play-it-safe NCI. But it may be a mistake to believe those studies would have been funded by a take-a-risk NCI, because we have the benefit of hindsight and a great deal of forgetting. Before the research was carried out — i.e., at the time it would have been a grant proposal — every one of those would-be-breakthrough proposals would have looked just as promising as a dozen of their contemporaries that turned out to be dead-ends and are now lost to history. So it’s not at all clear that all of those breakthroughs would have been funded within a system that took bigger risks, because they would have been competing against an even larger pool of equally (un)promising high-risk ideas.

3. The political problem is that even if we could solve #1 and #2, we as a society would have to have the stomach for putting up with a lot of research that produces no meaningful results. The scientific community, politicians, and the general public would have to be willing to constantly remind themselves that scientific dead ends are not a “waste” of research dollars — they are the inevitable consequence of taking risks. There would surely be resistance, especially at the political level.

So what’s the solution? I’m sure there could be some improvements made within the current system, especially in getting review panels and program officers to reorient to higher-risk studies. But I think the bigger issue has to do with the overall amount of money available. As the top-rated commenter on Kolata’s article points out, the FY 2010 defense appropriation is more than 6 times what we have spent at NCI since Nixon declared a “war” on cancer 38 years ago. If you make resources scarce, of course you’re going to make people cautious about how they invest those resources. There’s a reason angel investors are invariably multi-millionnaires. If you want to inspire the scientific equivalent of angel investing, then the people giving out the money are going to have to feel like they’ve got enough money to take risks with.

On cultural significance and the value of a life

With Michael Jackson and Farrah Fawcett dying on the same day, there are a lot of articles discussing them together. This one at MSNBC is a pretty representative example.

In reading the coverage, I can’t help but think that Farrah Fawcett’s cultural significance is getting pumped up. Not to say that she wasn’t a major cultural icon. But I think there’s something else going on.

As a culture we like to think that the value of a life is unmeasurable, and therefore all lives are equally sacred (economists be damned). Nobody would say that the extent to which society publicly mourns somebody’s death is a measure of their worth as a human being (most of us don’t get TV specials when we die). Media coverage is a function of fame and public impact, and private funerals are about mourning a beloved person, and those are usually completely different spheres. But the fact that Farrah Fawcett and Michael Jackson died on the day puts us in the uncomfortable position of looking at their deaths side-by-side. Fame and human worth get mixed together in the media coverage of somebody who has just died, and it’s hard to only apply one standard and not the other.

In this case, if we step back and look objectively in terms of cultural significance, I don’t think it’s hard to reach the conclusion that Farrah Fawcett and Michael Jackson were not on the same level. That isn’t to diminish the place that Fawcett held in society. But few people in history could measure up to Michael Jackson, who triggered a tectonic shift in how our culture thinks about music, dance, race, and celebrity. Rationally we can acknowledge that inequality without implying that one person’s life was more valuable than the other’s. But I suspect that on a gut level, it feels vaguely ghoulish to do so too loudly. So the end result is that Fawcett may be getting credited for even greater cultural significance than she otherwise would have.

(Related tangent: I can’t be the only one who feels uncomfortable every year during the Oscar tributes to Hollywood folks who’ve passed away, seeing the famous actors get louder applause than the obscure cinematographers. I suspect it’s the same sort of conflict between fame vs. human worth that’s driving that discomfort.)

Evidence-based policy

I’m all for basing social policy on good social science evidence. But as Dean Dad writes:

We have anecdotal evidence that suggests that students who actually take math for all four years of high school do better in math here than those who don’t. We also have anecdotal evidence that bears crap in the woods. Why the hell do the high schools only require two years of math?

I say we can bypass the regression analysis on this one.

Taking aim at evolutionary psychology

Sharon Begley has a doozy of an article in Newsweek taking aim at evolutionary psychology. The article is a real mixed bag and is already starting to generate vigorous rebuttals.

As background, the term “evolutionary psychology” tends to confuse outsiders because it sounds like a catchall for any approach to psychology that incorporates evolutionary theory and principles. But that’s not how it’s used by insiders. Rather, evolutionary psychology (EP) refers to one specific way (of many) of thinking about evolution and human behavior. (This article by Eric Alden Smith contrasts EP with other evolutionary approaches.) EP can be differentiated from other evolutionary approaches on at least 3 different levels. There are the core scientific propositions, assumptions, and methods that EPs use. There are the particular topics and conclusions that EP has most commonly been associated with. And there is a layer of politics and extra-scientific discourse regarding how EP is discussed and interpreted by its proponents, its critics, and the media.

Begley makes clear that EP is not the only way of applying evolutionary principles to understanding human behavior. (In particular, she contrasts it with human behavioral ecology). Thus, hopefully most readers won’t take this as a ding on evolutionary theory broadly speaking. But unfortunately, she cherrypicks her examples and conflates the controversies at different levels — something that I suspect is going to drive the EP folks nuts.

At the core scientific level, one of the fundamental debates is over modularity versus flexibility. EP posits that the ancestral environment presented our forebears with specific adaptive problems that were repeated over multiple generations, and as a result we evolved specialized cognitive modules that help us solve those problems. Leda Cosmides’s work on cheater detection is an example of this — she has proposed that humans have specialized cognitive mechanisms for detecting when somebody isn’t holding up their obligations in a social exchange. Critics of EP argue that our ancestors faced a wide and unpredictable range of adaptive problems, and as a result our minds are more flexible — for example they say that we detect cheaters by applying a general capacity for reasoning, not through specialized cheater-detecting skills. This is an important, serious scientific debate with broad implications.

Begley discusses the modularity versus flexibility debate — and if her article stuck to the deep scientific issues, it could be a great piece of science journalism. But it is telling what topics and examples she uses to flesh out her arguments. Cosmides’s work on cheater detection would have been a great topic to focus on: Cosmides has found support across multiple methods and levels of analysis, and at the same time critics like David Buller have presented serious challenges. That could have made for a thoughtful but still dramatic presentation. But Begley never mentions cheater detection. Instead, she picks examples of proposed adaptations that (a) have icky overtones, like rape or the abuse of stepchildren; and (b) do not have widespread support even among EPs. (Daly and Wilson, the researchers who originally suggested that stepchild abuse might be an adaptation, no longer believe that the evidence supports that conclusion.) Begley wants to leave readers with the impression that EP claims are falling apart left and right because of fundamental flaws in the underlying principles (as opposed to narrower instances of particular arguments or evidence falling through). To make her case, she cherrypicks the weakest and most controversial claims. She never mentions less-controversial EP research on topics like decision-making, emotions, group dynamics, etc.

Probably the ugliest part of the article is the way that Begley worms ad hominem attacks into her treatment of the science, and then accuses EPs of changing topics when they defend themselves. A major point of Begley’s is that EP is used to justify horrific behavior like infidelity, rape, and child abuse. Maybe the findings are sometimes used that way — but in my experience that is almost never done by the scientists themselves, who are well aware of the difference between “is” and “ought.” (If Begley wants to call somebody out on committing the naturalistic fallacy, she should be taking aim at mass media, not science.) Begley also seems to play a rhetorical “I’m not touching you” baiting game. Introducing EP research on jealousy she writes, “Let’s not speculate on the motives that (mostly male) evolutionary psychologists might have in asserting that their wives are programmed to not really care if they sleep around…” Then amazingly a few paragraphs later she writes, “Evolutionary psychologists have moved the battle from science, where they are on shaky ground, to ideology, where bluster and name-calling can be quite successful.” Whahuh? Who’s moving what battle now?

The whole thing is really unfortunate, because evolutionary psychology deserves serious attention by serious science journalists (which Begley can sometimes be). David Buller’s critique a few years ago raised some provocative challenges and earned equally sharp rebuttals, and the back-and-forth continues to reverberate. That makes for a potentially gripping story. And EP claims frequently get breathless coverage and oversimplified interpretations in the mass media, so a nuanced and thoughtful treatment of the science (with maybe a little media criticism thrown in) would play a needed corrective role. I’m no EP partisan — I tend to take EP on a claim-by-claim basis, and I find the evidence for some EP conclusions to be compelling and others poorly supported. I just wish the public was getting a more informative and more scientifically grounded view of the facts and controversies.

The monkey game

Sometimes when I’m in a public place, I like to amuse myself by playing what I call the “monkey game.” I look at people going by and try to observe them as if I were a primatologist. I ignore what they’re saying (and even the very fact that their verbalizations have any meaning at all) and set aside any sophisticated human psychological concepts. Instead, I try to interpret their behavior by putting it into a few basic categories that an ethologist might use when observing any social species: dominance displays, grooming, alliance-building, territoriality, various mating-related behaviors (attracting, maintaining, guarding), etc.

It’s amazing how much behavior you can make sense of with this sort of reverse anthropomorphizing. Try it next time you’re at the mall.

How a sexist environment affects women in engineering

Women in traditionally male-dominated fields like math and engineering face the extra burden that their performance, beyond reflecting on them individually, might be taken as broader confirmation of stereotypes if they perform poorly. A newly published series of experiments by Christine Logel and colleagues tested the effects of such stereotype threat among engineering students.

Standardized observations showed that male engineering students who had previously expressed subtle sexist attitudes on a pretest were more likely, when talking with a female engineering student about work issues, to adopt a domineering posture and to display signs of sexual interest (such as noticeably looking at the woman’s body).

In the next 2 experiments, female engineering students were randomly assigned in one experiment to interact with males who had endorsed different levels of subtle sexism, and in a second experiment with an actor who randomly either displayed or did not display the domineering/sexual nonverbal behaviors. Women performed worse on an engineering test after interacting with the randomly-assigned sexist males (or males simulating sexists’ nonverbal behavior).

In another experiment, women’s poorer performance was shown to be limited to stereotype-related tests, not a broad cognitive deficit. In a final experiment, interacting with a domineering/sexually interested male caused women to have temporarily elevated concern about negative stereotypes, which they subsequent attempted to suppress (thought suppression being a well-known resource hog).

The results further support the idea that even subtle sexism can be toxic in workplace environments where women are traditionally targets of discrimination.

Statistics terms that would make good metal band names, Part 2

The Five-Factor Solution.

Statistics terms that would make good metal band names, Part 1

Variance decomposition.