The Decline Effect

[n2doc]

THE TRUTH WEARS OFFIs there something wrong with the scientific method?
by Jonah Lehrer
DECEMBER 13, 2010

n September 18, 2007, a few dozen neuroscientists, psychiatrists, and drug-company executives gathered in a hotel conference room in Brussels to hear some startling news. It had to do with a class of drugs known as atypical or second-generation antipsychotics, which came on the market in the early nineties. The drugs, sold under brand names such as Abilify, Seroquel, and Zyprexa, had been tested on schizophrenics in several large clinical trials, all of which had demonstrated a dramatic decrease in the subjects’ psychiatric symptoms. As a result, second-generation antipsychotics had become one of the fastest-growing and most profitable pharmaceutical classes. By 2001, Eli Lilly’s Zyprexa was generating more revenue than Prozac. It remains the company’s top-selling drug.

But the data presented at the Brussels meeting made it clear that something strange was happening: the therapeutic power of the drugs appeared to be steadily waning. A recent study showed an effect that was less than half of that documented in the first trials, in the early nineteen-nineties. Many researchers began to argue that the expensive pharmaceuticals weren’t any better than first-generation antipsychotics, which have been in use since the fifties. “In fact, sometimes they now look even worse,” John Davis, a professor of psychiatry at the University of Illinois at Chicago, told me.

Before the effectiveness of a drug can be confirmed, it must be tested and tested again. Different scientists in different labs need to repeat the protocols and publish their results. The test of replicability, as it’s known, is the foundation of modern research. Replicability is how the community enforces itself. It’s a safeguard for the creep of subjectivity. Most of the time, scientists know what results they want, and that can influence the results they get. The premise of replicability is that the scientific community can correct for these flaws.

But now all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain. It’s as if our facts were losing their truth: claims that have been enshrined in textbooks are suddenly unprovable. This phenomenon doesn’t yet have an official name, but it’s occurring across a wide range of fields, from psychology to ecology. In the field of medicine, the phenomenon seems extremely widespread, affecting not only antipsychotics but also therapies ranging from cardiac stents to Vitamin E and antidepressants: Davis has a forthcoming analysis demonstrating that the efficacy of antidepressants has gone down as much as threefold in recent decades.



Read more http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer

[xchrom]

1. i'm rec-ing this -- because i'm both skeptical and curious. nt

[HereSince1628]

2. Interesting article. Noise has value when it yields a positive result

Then along comes follow on studies and competitors who want to bust an accepted premise to pieces and the story changes...

Still, it seems to me that this is not so much a decay of -real- truth as it is the decay of inescapable truthiness that clouds humans engaged in their pet projects.

Now, if THAT result holds up I may continue to find that reassuring rather than troubling! :rofl:

[SheilaT]

3. My very first thought it

to see if they are still be manufactured in the same place as originally. Has drug manufacture been outsourced to China yet?

[Jim__]

4. Fascinating. The impression I get from the article is that this is most likely due to ...

... confirmation bias. We see what we want to see and disregard the rest. If that's the case, I don't see any easy way of addressing this.

[Igel]

5. There is no great way of addressing confirmation bias.

It's why intellectual diversity and academic freedom, for all their pitfalls, are Good Things.

It's likely that the early studies for medication looked at "good" subjects chosen because they fit the narrowly constructed criteria, and the criteria, dosing, etc., etc. were well monitored and enforced. Then once the med is approved the physicians say "screw the criteria" so a wider range of people receive the drug; the dosing, etc., is less stringently monitored and enforced.

So assuming that the initial study results are reported accurately and the experiment done as described, you have the problem that the later replications aren't actually strict replications. They're sort of cheap knock-offs.

Education and linguistics are rife with this kind of problem. I assume it's fairly common in fields that lack a certain kind of "protocol rigor." You see educational studies that yield great results with the 30 black kids from a specific 4 square blocks of a single city with specially trained and highly motivated teachers in specially organized classrooms with specially trained counselors meeting with the volunteer and already-motivated parents on a strict schedule, all supervised by a highly motivated and supportive administration. It works like gang-busters. Then you publish it and the "program" is used by people who looked at it for 5 minutes and decided to change nothing except the name of the program they were using; the classrooms don't change, the administration barely cares as long as the right-named program is touted and the parents are at best barely aware of what the program is and there's no additional money for new materials. The program flops and you're left with the public wondering what kind of a git you are, wasting their time and money.

[dimbear]

7. Two possibilities: one is that the drugs which performed best

were selected and retained even tho their superiority in the tests was partly due to chance. You see this in selective breeding, the offspring of tall parents are taller than average but not as tall as their parents because height is partly due to 'noise.'

Alternative two, the effects of inflation on money in general making it less valuable to the researchers to fake the tests.

[skoalyman]

6. its a terrable thing to have to go through.

My mom had her first nervous breakdown in the early nineties,She did good on her meds for couple years, then she relapsed ten years later and has never been the same. We just got her out of the hospital were they tried different meds but they don't do anything but sedate her, or just barely takes the edge off for a couple of hours.

[Jim Lane]

8. Another possible explanation: increasing placebo effect

I think I read somewhere else that, perhaps because the public is becoming more confident in pharmaceutical solutions generally, placebo effects are becoming stronger. If so, it could be that the drugs are just as effective as ever, but the placebos given to the control group are more effective than they used to be. The result would be that the improvement brought about by the real drugs would be less.

Remember that, in a double-blind trial, experimental drugs aren't compared with a known no-drug regimen. Instead, the experimental group gets the drug and the control group gets the placebo (an inert pill known to have no effects). Some people who get a pill will get better because they think they will or because they're comforted that someone is doing something to help them. The double-blind trial is to control for those effects.

[n2doc]

9. Some commentary

Category: Communicating science • Skepticism
Posted on: December 30, 2010 5:30 PM, by PZ Myers

eople keep sending me this link to an article by Jonah Lehrer in the New Yorker: The Decline Effect and the Scientific Method, which has the subheadings of "The Truth Wears Off" and "Is there something wrong with the scientific method?" Some of my correspondents sound rather distraught, like they're concerned that science is breaking down and collapsing; a few, creationists mainly, are crowing over it and telling me they knew we couldn't know anything all along (but then, how did they know…no, let's not dive down that rabbit hole).

I read it. I was unimpressed with the overselling of the flaws in the science, but actually quite impressed with the article as an example of psychological manipulation.

The problem described is straightforward: many statistical results from scientific studies that showed great significance early in the analysis are less and less robust in later studies. For instance, a pharmaceutical company may release a new drug with great fanfare that showed extremely promising results in clinical trials, and then later, when numbers from its use in the general public trickle back, shows much smaller effects. Or a scientific observation of mate choice in swallows may first show a clear preference for symmetry, but as time passes and more species are examined or the same species is re-examined, the effect seems to fade.

This isn't surprising at all. It's what we expect, and there are many very good reasons for the shift.

more
http://scienceblogs.com/pharyngula/2010/12/science_is_not_dead.php