Facts Losing Their Truth

A key tenet for data-driven marketing professionals is fact-based decision-making. However, something strange is occurring in scientific studies, where tested and proven results are becoming difficult to replicate. The challenge for marketing professionals is to realize that what is true today may not be true tomorrow.

Jonah Lehrer, in the December 13, 2010 issue of the New Yorker, authors a stunning article on the increasing ineffectiveness of therapeutic drugs. Medications tested eight to ten years ago on large populations seem to work only half as well in similar trials today. So then, many tests for medications in randomized, double blind clinical trials (considered the gold standard in science) are not repeatable—not even close! Lehrer points out the conundrum: “The test of replicability … is the foundation of modern research. Replicability is how the (scientific) community enforces itself.”

Lehrer mentions this phenomenon isn’t simply relegated to pharmaceuticals. In fact, in vitamin and cardiac stent tests, what was once “known” and proven is now mostly without validation if tested again. “It’s as if our facts were losing their truth,” Lehrer says. “Claims that have been enshrined in textbooks are suddenly unprovable.”

Observation, testing, experimentation, and replication are foundations of scientific process. The issue in a nutshell is explained by Lehrer: “If replication is what separates the rigor of science from the squishiness of pseudo-science, where do we put all these rigorously validated findings that can no longer be proved?”

Some may argue that a simple case of changing populations, or even improvements in psychology account for changes in testing results. Or even that perhaps there was miscalculation, statistical fluke, or other anomaly in previous trials. These are entirely possible explanations, but it is unlikely that hundreds, if not thousands of studies, would be facing the same challenges today. And even so, there might be some statistical difference, but variances of 30-50% are common in repeated experiments, especially those conducted over a period of years. There must be something to this drastic decline.

The “decline effect” is keeping scientists across the globe quiet about whether their experiments can be replicated. One scientist was told by his mentor that his “real mistake was trying to replicate his work” and that he would only set himself up for “disappointment.”

Adding insult to injury, it appears that scientific journals don’t want to publish findings regarding the decline effect. Biologist Leigh Simmons from the University of Western Australia tried to submit results to various publications detailing his difficulty in duplicating his experiments, but “the journals only wanted confirming data,” he says. And that’s ultimately a shame because there is much to learn not just from success, but failure.

This then, may explain why much scientific “evidence” is often contradictory. One year, we learn that coffee is good for health, and then two years later, it’s bad. We find out that hormone replacement therapy for a menopausal woman is recommended one year, only to find out the next that it never should have been recommended in the first place. “The situation is even worse when the subject is fashionable,” Lehrer says.

All told, there is bias, conflict of interest, faulty design, selective publishing, psychology progress and certainly other developments at work. However, the decline effect is a much more significant finding as it “reminds us how difficult it is to prove anything.” Because what is true today may not be true tomorrow, the best result when dealing with “proven” scientific facts is a healthy dose of skepticism and a question everything mindset.

Questions:

  • If yesterday’s rigorous and validated findings can no longer be proved, what does this mean for those practicing empirical, fact based decisioning?
  • Statistically speaking, the decline effect shouldn’t be happening, but it does. What other reasons explain why facts can lose their “truthiness” over time?
Advertisements

2 comments

  1. This was brave. Now you might get excorsised from the scientific community. 🙂

    >>> If yesterday’s rigorous and validated findings can no longer be proved, what does this mean for those practicing empirical, fact based decisioning?

    The same thing as it did in the past and you have mentioned really well: question everything. Think of why your results are what they are. Interpretation is still one of the key mistakes I see.

    >>> Statistically speaking, the decline effect shouldn’t be happening, but it does. What other reasons explain why facts can lose their “truthiness” over time?

    I was studiing “Satisfaction” for my masters work. And there’s like zillions of researches done on that subject. Zillions of definitions and all ending up with un-replicability. I was struggling with it for a long time. Felt that there is something missing. So I had really big trouble time to make interpretations of my findings. Until one day I’ve noticed that we’re all studiing the non-existing thing. That satisfaction itself is actually just a word which describes a lot of things. Then I started modelling this and came out with something that made my findings valid and interpretation most probably replicable.

    So in my little opinion, knowing your data, research model and interpretation is not done correctly. There’s way too many researchers out there that make money out of knowing the statistical methods or just their “title” (dr., prof.,…).

    —–

    For a little ending, I can see this happening now in the “web site optimisation” community. They take the facts from some research and everyone makes the same websites then, because it was proven in that research that a big red button right-up is the key to success. I’m not saying that it is not. I’m just questioning if this is the only possible solution. 🙂

    • Dusan, You know how I like to challenge the status quo. When I find a good article like the NewYorker on “facts losing their truth”, I just have to write about it.

      To your point on SEO, that when we get dogmatic about a particular “fact”, that’s when we’re in trouble. The decline effect is noticed in just about every scientific study and that’s why these findings are so important. The truth loses it’s “truthiness” over time.
      Thank you for commenting. Appreciate the friendship we’ve developed over the last few years. Blessings!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s