I’ve quoted some of this before, but it was buried in a long post and it’s worth quoting at greater length and on its own. It succinctly lays out his views on several issues relevant to present-day discussions of replication in science. Specifically, Popper makes clear that (1) scientists should replicate their own experiments; (2) scientists should be able to instruct other experts how to reproduce their experiments and get the same results; and (3) establishing the reproducibility of experiments (“direct replication” in the parlance of our times) is a necessary precursor for all the other things you do to construct and test theories.
Kant was perhaps the first to realize that the objectivity of scientific statements is closely connected with the construction of theories — with the use of hypotheses and universal statements. Only when certain events recur in accordance with rules or regularities, as is the case with repeatable experiments, can our observations be tested — in principle — by anyone. We do not take even our own observations quite seriously, or accept them as scientific observations, until we have repeated and tested them. Only by such repetitions can we convince ourselves that we are not dealing with a mere isolated ‘coincidence’, but with events which, on account of their regularity and reproducibility, are in principle inter-subjectively testable.
Every experimental physicist knows those surprising and inexplicable apparent ‘effects’ which in his laboratory can perhaps even be reproduced for some time, but which finally disappear without trace. Of course, no physicist would say in such a case that he had made a scientific discovery (though he might try to rearrange his experiments so as to make the effect reproducible). Indeed the scientifically significant physical effect may be defined as that which can be regularly reproduced by anyone who carries out the appropriate experiment in the way prescribed. No serious physicist would offer for publication, as a scientific discovery, any such ‘occult effect,’ as I propose to call it — one for whose reproduction he could give no instructions. The ‘discovery’ would be only too soon rejected as chimerical, simply because attempts to test it would lead to negative results. (It follows that any controversy over the question whether events which are in principle unrepeatable and unique ever do occur cannot be decided by science: it would be a metaphysical controversy.)
– Karl Popper (1959/2002), The Logic of Scientific Discovery, pp. 23-24.
The Hardest Science 
Great quote! Though, by “anyone”, I sure hope Popper didn’t mean me. I wouldn’t have the slightest idea how to turn on a Large Hadron Collider, let alone find a Higgs boson :)
Same with me and MRI. I mean, it took me forever to find the off switch when I was done. Hey, why are you all looking at me like that?
Joking aside, I read “anyone who carries out the appropriate experiment in the way prescribed” as allowing that it might involve substantial training and expertise. Just that it must be possible to communicate that.
I agree completely. We need to do a better job of communicating these details and training the right expertise. By we, I largely mean me.
That said, I still don’t know if I personally have the right aptitude to acquire the necessary expertise in some fields. For example, I just don’t have the stomach for surgery.
I agree with the sentiment that replication is crucial to good science, but I bet very few people would be willing to be as strict as Popper about failures to replicate (and failures to confirm a prediction). If they were, no theory would last very long. As soon as one failed to replicate a result (or failed to confirm a prediction, in general) fulfilled, the theory would have to be scrapped. If this ever happens on science, it is extremely rare. It seems that most of the time we assume the a theory is correct and that when predictions fail, something about the experiment or our background assumptions wasn’t right.
“…predictions are deduced from the theory….Next we seek a decision as regards these (and other) derived statements by comparing them with the results of…experiments. If this decision is positive, that is, if the singular conclusions turn out to be acceptable, or verified, then the theory has, for the time being, passed its test: we have found no reason to discard it. But if the decision is negative, or in other words, the conclusions have been falsified, then their falsification also falsifies the theory from which they were logically deduced” (1959, 27-33).
Right, there may be things besides the theory being tested that are required to deduce the predictions.
That is, T & B -> P. So then ~P implies only ~T or ~B.
Related to the so-called Quine-Duhem thesis.
Thanks for your response.
I guess I am just wondering what the next step should be if Popper is right about replication. We agree that Popper’s view is too simple. So we depart from him. So I am wondering what you take to be the proper response to failed replications. It sounds like you accept the Duhem-Quine disjunction that results from failed replications: either the theory is false or some unknown set of background assumptions is false. If this is right, then it seems that a sensible response to failed replications is to begin testing background assumptions? Is that right? And, in your view, is this what tends to happen in psychology?
(This is assuming, of course, that there isn’t obvious evidence of, say, fabricated data or obvious errors in analysis, which might imply that there is little need to test background assumptions).
Nick wrote: “it seems that a sensible response to failed replications is to begin testing background assumptions”
I think that’s exactly right. If Lab A reliably gets one result and Lab B reliably gets another, that opens up new avenues for investigation and potentially new and interesting discoveries. I talked about this in a previous blog post:
https://hardsci.wordpress.com/2014/07/01/some-thoughts-on-replication-and-falsifiability-is-this-a-chance-to-do-better/
I think this is conflating 2 things. The passage I quoted is from a section on objectivity. Popper was saying that scientific evidence needs to be objective, and the way we know that it is objective is that it’s inter-subjectively testable – anyone who does the right experiment in the right way can observe the result. So establishing the reproducibility of evidence is a precursor to theory-testing. If evidence is not reproducible, then it has no bearing on theory-testing one way or the other.
The second issue, which only comes later, is what to do when we find (reproducible) evidence that contradicts a theory. Some people read Popper as being too willing to abandon theories. I think he’s more nuanced. But in any case, I think Lakatos’s extension of Popper, and the distinction between progressive vs. degenerative amendments to theories, addresses this well (and the related issues from Duhem-Quine). Meehl’s discussion of Lakatos is excellent and should be required reading for psychologists:
Click to access 147AppraisingAmending.pdf
Thanks for this response as well. I agree that there are two separate points here. Sorry I didn’t make that clearer in my comments.
And thanks for the references. I am familiar with Lakatos. I will look into Meehl.