Further thoughts on replication and continuity in science
I’d like to follow up a little further on the issues I raised regarding the replication movement in science, particularly psychology, in my last post. A lot of this bears on the issues about the nature of science and the degree to which behavioral sciences in general and psychology in particular qualify as sciences. I’ve discussed this fairly extensively a couple of years ago (here and here for examples.)
When I said that replication is impossible, I certainly didn’t mean to say that further research into any given problem shouldn’t be done. What I’m arguing against is the idea that one can replicate psychological experiments in the same way that all the 30 kids in the high school chem lab are replicating the production of oxygen and hydrogen through electrolysis of water. Do the same thing; get the same results, within a milliliter or two of error. That’s the way science works, isn’t it?
Well, yes, for certain kinds of science. Setting up the electrolysis experiment takes maybe half a dozen pieces of equipment, a few supplies, and a few procedures. Those items totally define the context and the circumstances of the experiment. In psychology on the other hand, as with any of the other behavioral sciences, the context is the Real World (see attached diagram), in which everything is connected to everything else in one way or another and it’s not possible to bound an inquiry such that much of the RW doesn’t affect it in one way or another.
Some have used this point to argue that psychology isn’t really a science at all. I think that’s wrong, because I think science is defined by a way of thinking and a disciplined way of inquiring into the universe, rather than by the immobility of the objects of its inquiry. Geology studies rocks; rocks are not materially affected by being studied. Psychology studies people; people are inherently part of the process of their being studied. But both geology and psychology operate within a framework of rules of inquiry, that govern the kind of inferences that one can draw from observations. Both responsible geologists and responsible psychologists are careful about the domain of generalizability of their findings. Where psychologists have gotten into trouble, it’s usually because they have been careless about extending this domain beyond its reasonable bounds.
As I said at the beginning, I’m strongly in favor of repeated inquiries into the same phenomenon, as well as comparing findings from these different studies. Where I think the replication movement goes off track is setting up an expectation that all these repeated inquiries should produce exactly the same findings, and that any study where a repeated kind of inquiry doesn’t generate exactly the same findings is somehow suspect. Given the shape of the RW (see above), it would be astounding if any did. Comparisons among studies claiming to examine the same phenomena can be very useful, particularly when one compares how they were carried out and the circumstances surrounding them. It’s certainly possible to work toward a research consensus about phenomena without requiring that each and every study produce exactly the same results.
