Sometimes remarkable effects found by scientific experiment start to seem exaggerated over time. Birds that had been believed to choose the most symmetrical mates turn out to be less selective than once thought. Drugs that were thought to be highly effective are later found to be less of a wonder than previously believed. This remarkable circumstance is called the decline effect.
Jonah Lehrer, a journalist with a degree in neuroscience and other extremely impressive credentials, wrote about the decline effect in the New Yorker and Wired. He sparked continuing debate about its implications.
Scientists disagree about what causes the decline effect, and about its potential effects on scientific study. Here are some things they name as possible causes of the decline effect:
Subject Selection
Initial drug studies often use carefully selected subjects who are most suitable for the drug undergoing testing. Later studies, and actual clinical use, are likely to use patients who are less than ideal, in circumstances that are less controlled. Therefore, the efficacy of the treatment may seem to decline.
Study Design
Faulty study design can also lead to results that fail to show such large effects when later researchers try to replicate them. Eager researchers may confound their own results in ways that researchers who follow them do not. The rewards for success are great, the penalties for error relatively small.
The Bandwagon Effect
In addition, initial sparkling results may draw in researchers persuaded that the new method will solve nagging problems better than ever. Their research may be unconsciously biased by their enthusiasm and desire to join the vanguard. Later studies may find flaws that these first adherents missed. A doctor who calls himself Orac on his blog is particularly perceptive about the Bandwagon effect, though he mostly disagrees with Mr. Lehrer.
Publication Bias
People prefer to hear, and bring, good news. Scientists are human, and have a bias towards the positive, the confirming, even the hopeful. In consequence, studies that support and enhance findings are more likely to see the light sooner than negative, disparaging studies.
Regression to the Mean.
In a small sample, a researcher may get a statistically extraordinary result, as a gambler playing roulette may hit four blacks in a row. Yet if a gambler plays a thousand times his or her results will aproximate half black and half red, minus the house edge. Where one result may deviate, a series of results will approach the norm. In other words, in small samples, an extraordinary result may be only a persuasive fluke.
Science strives for precision, but life is untidy. Though scientists are aware of possible sources of confounding errors, they still occur. Mathematical precision blurs where it meets the real world.
Humans seek certainty, though, and bedrock truths in a shifting world. Though truth is mutable, we grasp at it with religious fervor. What scientists seek are useful truths, even knowing that some truths are provisional.
It is likely that investigation of the decline effect will only strengthen researchers’ vigilance. Science still corrects itself, as scientists repeat and refine one another’s work. The scientific method is designed to expose error, through replication, and peer review, and experiment itself. People are by nature biased, stubborn, credulous joiners. It’s okay. Science is strong enough to take it.
Sources:
The Truth Wears Off by Jonah Lehrer http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer
Wired Q and A by Jonah Lehrer http://w ww.wired.com/wiredscience/2010/12/the-mysterious-decline-effect/
Is the Decline effect really so mysterious by “Orac” http://scienceblogs.com/insolence/2010/12/is_the_decline_effect_really_so_mysterio.php
