The Next Wave in Recommendation Systems?

While some internet privacy experts fret over use of cookies and web profiles for targeted advertising, the quest for personalization is about to go much deeper as web companies create new profiling techniques based on the science of influence.

Behavioral targeting on the web using cookies, http referrer data, registered user accounts and more is about to be significantly enhanced says columnist Eli Pariser.  In the May 2011 issue of Wired Magazine, in an article titled “Mind Reading”, Pariser discusses how website recommendation and targeting algorithms; “analyze our consumption patterns and use that information to figure out (what to pitch us next).”   However, Parser notes that the next chapter for recommendation systems is to discern the best approach in influencing online shoppers to buy.

In the article, Pariser cites an experiment by a doctoral student at Stanford where online shopping sites attempted to not only track clicks and items of interest, but also determine the best way to pitch a product. For example, pitches would alternate between an “Appeals to Authority”; as in someone you respect says you’ll like this product to “Social Proof”—everyone’s buying this product, so should you!

Taking a cue from the work completed by Dr. Robert Cialdini it appears that the next wave in recommendation algorithms is to learn our “decision triggers”, or the best way to persuade us to act. In his book “Influence: Science and Practice”, Cialdini documented six decision triggers of consistency, reciprocation, social proof, liking, authority and scarcity as mental shortcuts that help humans deal with the “richness and intricacy of the outside environment.”

Getting back to the Wired Magazine article, Eli Pariser says this means that websites will hone in on the best pitch for a particular online consumer and –if effective—continue to use it.  To illustrate this concept, Pariser says; “If you respond a few times to a 50% off in the next ten minutes deal, you could find yourself surfing a web filled with blaring red headlines and countdown clocks.”

Of course, shoppers buy in various ways and not always in the same manner. However, the work of Robert Cialdini shows that in the messy and complicated lives of most consumers that mental shortcuts help with the daily deluge of information. Therefore, this new approach of recommendation systems using principles of psychology in tailoring the right way to “pitch” online shoppers, might just work.

There’s no doubt that recommendation systems already take into account principles of social proof and liking, but there’s a lot more room for improvement, especially other areas that Cialdini has researched. The answer to ‘why we buy’ is about to be taken to a whole new level.


  • What’s your take on this next development in recommendation systems? Benefit or too much “Big Brother”?
  • Are you moved by “act now” exhortations? What persuasion technique/s work best on you?

Eight Things You Should Know About Statistics

Statistics have been called “an engine of knowledge” by one risk management expert. And while it’s true that some business managers don’t have a fundamental grasp of statistical concepts, we also know there is opportunity for misuse of mathematics. Is statistics the “new grammar” or are efforts to attach certainty to life’s events doing more harm than good?

In May 2010’s issue of Wired Magazine, author Clive Thompson laments the poor mathematical literacy of his fellow citizens. For example, he cites people laughing at the concept of global warming as they face some of the harsher winters on record, or the extra-vocal debate on vaccines and possible links to autism. Mr. Thompson would tell us that it’s the trend lines that matter, and we too often look at the trees and miss the forest.

The problem, he says, is that “statistics is hard” and an overall understanding of this important discipline is severely lacking. He says, “If you don’t understand statistics, you don’t know what’s going on, and you can’t tell when you’re being lied to.”

Thompson is correct that statistics are difficult for most of us, and that thinking by the numbers takes training and much effort. It’s also true that one must understand statistical concepts, especially when percentages, populations, and probabilities are bandied about in business and technical press. However, broader acceptance of the power of statistics should be tempered with limitations of this mathematical science.

Before accepting any statistic, study or experiment as gospel, the following should be considered (there may be more…):

1. Assumptions: What are the assumptions underpinning the research? As seen from recent debate on CBO numbers for the U.S. health reform package, assumptions matter tremendously.
2. History: How much historical data was used in the study? What was the time scale? As seen from the 2008 financial crisis, the models used by Wall Street mavens often only took into account 10 years of data in judging the volatility and probability of failure of complex financial instruments.
3. Samples: Are the samples selected randomly? From what populations? Is there enough data for statistical significance?
4. Data Quality: The output is only going to be as good as the quality of data feeding the analysis. Garbage in, garbage out.
5. Survivorship Bias: Author Nassim Taleb points out that “losers are often not in the sample.” Does the analysis include a population of survivors and those who also failed?
6. Falsification and Omission: Yes, in an era of IPCC’s Climate Gate, one needs to ascertain if data are hidden, missing or outliers ignored.
7. Association equals causation fallacy: Correlation does not equate to causation (a common mistake made by marketing and finance executives alike).
8. Proper Application of Statistics: The effective use of statistics by insurance actuaries, scientists, and even casino managers is well-documented. However, real danger results when mathematical concepts are used to denote certainty indecision-making and divining behavior of markets.

Now, please don’t get me wrong. Statistical analysis is very important for many industries (e.g., health care, transportation, and manufacturing). Statistics, however, can give us an illusion of control in a world that’s much more complex than our models suggest. Nassim Taleb, author of the Black Swan likes to remind us that “(real) life isn’t a casino.”

Statistical analysis is definitely a powerful gadget in the business manager’s decision-making toolkit. But one needs to understand the limitations of this science.

After all, Taleb points out that many of today’s statistical models work as though we have “full knowledge of the probability of future outcomes.” And this just isn’t so, especially when it comes to fat tails, or the “ten sigma” event. Indeed, sometimes those rare events have extremely large impacts. Were he alive today, the former captain of the Titanic, E.J. Smith would wholeheartedly agree.

• Clive Thompson calls statistics “the language of data.” How important is it for marketers to understand and apply statistical concepts?
• “Lies, damned lies, and statistics” is a phrase popularized by Mark Twain in the context of using statistics to unduly persuade, obfuscate or even swindle. Can statistics get its reputation back? If so, how?