Why Predicting the Future is So Darn Difficult

Predicting the future is difficult—just ask George Soros. While Soros is often celebrated for the $1 billion profit he made in 1992 on a bet that the pound sterling would collapse in valuation, other trades ended up costing him almost as much money as he made.

As detailed in Sebastian Mallaby’s “More Money than God”, leading up to October 19, 1987, Soros’ Quantum fund had been up 60%. However, when “Black Monday” hit and the Dow Jones lost 22.6% of its value, Soros was in the middle of the mess deciding whether to sell or buy. While he held his positions through Wednesday of that week, on Thursday he abruptly changed his mind and sold positions worth $1 billion. Soros’ decision to unload his massive portfolio sparked other traders to also sell stocks and bonds, thus causing a downward spiral in markets. At the end of the day, Soros was out of the market; however his Quantum fund lost $840 million!

Alas, that’s the problem with gut decision making, you say. Soros should have used quantitative analysis, right? Even quantitative analysis can produce the wrong outcome.

According to Roger Lowenstein’s “When Genius Failed”, hedge fund Long Term Capital Management (LTCM) was chock full of the best minds in finance. Assembling PhDs in finance, mathematics, economics and more, LTCM partners built sophisticated trading models based on the assumption that while investors sometimes panic or get too optimistic, eventually markets settle towards equilibrium. And in moments of panic or too much optimism, LTCM’s partners believed there was money to be made.

Unfortunately, LTCM is a case study in over reliance on analytical models for decision making. Lowenstein writes, “LTCM Partners believed that all else being equal, the future would look like the past” and this—of course—turned out to be a calamitous assumption. LTCM bet heavily on models, often doubling down on investments that they believed had an infinitesimal probability of failing. The assumption underpinning these models was that markets are efficient and rational. And when markets proved otherwise, Lowenstein notes, “The fund with the highest IQs lost 77% of its capital, while the ordinary stock investor doubled his money during the same period.”

It is apparent in studying Soros and LTCM, that even the most experienced minds supplemented by analytical tools and techniques can make extremely poor decisions about the future. So why is predicting the future so difficult?

In Scientific American, author Michael Shermer has an answer. He says that the world is a “messy, complex and contingent place with countless intervening variables and confounding factors which our brains are not equipped to evaluate.” He says we should stick to short term predictions rather than those longer term trends which we so often get wrong.

Does this mean that any attempt to predict the future is for naught? Of course not, as there are definitely limited applications for prediction models in preventing fraud, recommending products, discerning customer defections, and more. Even three day weather forecasts are more right than wrong!

The real lesson is that predicting the future is hard, especially when we’re confronted with millions of potential variables. Deciding which variables to pay attention to, and weighting the importance of those variables is especially difficult.

And maybe then, the solution is not so much to spend countless hours predicting the future (especially for strategic decisions), but instead to expend energy preparing for it.


* Do you agree with Michael Shermer that our brains are not equipped to evaluate our “messy world”?
* What other case studies have you encountered where either gut or analytical decision making was taken to the extreme?

Why Capacity Management Matters to Business Executives

bucketIn a challenging global slowdown, the world seems awash in capacity. Scans of major business publications show airlines reducing flights, companies furloughing or firing employees, and manufacturers closing plants. If you agree that it appears there is more unused capacity than demand, why should capacity management matter?

It would seem in the “Great Recession,” capacity planning and management should be a minimal consideration. In fact, capacity utilization for many industries is at an all time low.

For example, from January 2008 to January 2009, according to the Financial Times, the demand for automobiles in the United States fell from 15.9 million to 9.6 million per year. And Wall Street Journal reports Federal Reserve Chairman Ben Bernanke told the House Budget Committee recently, “The slack in resource utilization remains sizable”.

As companies attempt to cope with a “new normal,” painful restructuring processes have included reducing “capacity” in human resources, plants and equipment, information technology, number of brands, distribution channels, and even debt covenants. All this restructuring is intended to pare down capabilities to what is perceived as a new reality in market conditions.

Indeed, observing macro-economic conditions, it’s tempting to write off “growth.” However, “growth” is far from dead.

Take for example, the exponential growth trends of Facebook and Twitter. In January 2009, Facebook touted its 150 millionth user, and in May 2009 surpassed 225 million users! One site projects Facebook to have 300 million users by the end of the year. Twitter’s growth has also been phenomenal—audiences grew 40% in just 30 days (March-April 2009).

In fact, “growth” exists (often exponentially) in areas such as data volumes, populations, energy usage, Moore’s Law, GDPs of select countries (India, China etc), education expenditures, and unfortunately—state and national debts!

Growth also can be found in micro-segments and categories such as increases in market share of private label brands vs. national brands on grocery store shelves, or Apple’s share of the smartphone market. Once our eyes are opened to growth trends, it’s quite easy to see signs of expansion everywhere!

The ability to meet the needs of your customers now and in the future is a critical function of any business. That’s what capacity management is all about. Spikes in demand could mean that your company is leaving money on the table and/or failing to meet customer needs. Need proof? For customer reaction, simply perform a web search on keywords “Twitter down time” or “Twitter outage” and you’ll gain evidence of how important capacity management really is.

Indeed, capacity management isn’t a one-time, annual event. It should be a continual process of making sure your business can scale up or down to meet customer needs. With a thumb on the pulse of demand, marketers have a responsibility to help establish a well documented capacity plan and process that considers future business requests.

Sound like simple, common sense, right? Properly predicting demand is anything but easy. Considerations must include a clean and accurate set of historical data, an analytical infrastructure to compute and analyze data, an understanding of the current state of the business and its capabilities, future growth projections based on applicable trends, and then a gap analysis of what it would take to scale based on various “what-if” scenarios.

Capacity management is all about reducing surprises. Take a good, hard look at your business. What’s growing? Something surely is.

What marketing campaigns are you preparing? What happens—for goodness sake—if they’re too successful and demand exceeds available supply? McDonald’s in India had to scale back marketing campaigns for Chicken McNuggets because they couldn’t keep up with demand. Good marketing is making promises your company CAN deliver.

Can you accurately predict if and when you’ll run out of resources to meet customer needs? Can you afford not to properly manage “capacity”?


• There appears to be a glut of capacity worldwide (i.e. shipping, telecommunications, manufacturing etc.). Should marketers be concerned with the concept of capacity management?
• What are the ramifications of getting capacity management wrong?
• Businesses are adding flexibility to meet spikes in demand through vehicles like cloud computing, temporary labor and outsourcing. Can you think of others?
• Suppose “capacity management” is built into the function of an annual strategic planning exercise. What might be a pitfall of this approach?

Mathematics and Marketing

math symbolsThink marketing doesn’t have much to do with mathematics?

Mathematics  is giving some companies a competitive edge in better understanding customers. Indeed, companies across all industries are now capturing data and creating rich profiles of customers to “predict” their wants, needs and future desires.  Mathematics has left the ivory tower of academia for a marketing department near you. Are you ready for this massive paradigm shift?

Let’s be clear as one who has worked for many global consulting companies,  I hate the phrase, “paradigm shift”. The words are close to meaningless due to overuse.

However, in this rare instance, where the world of mathematics is invading the marketing kingdom, it makes sense to emphasize a new way of thinking that is radically changing the way marketers do business.

Marketers have always wanted to know more about customers—after all, better segmentation and targeting of a customer base helps improve marketing ROI and ultimately increases satisfaction as customers are not bombarded with irrelevant offers.

Fortunately for marketers, advances in technology (both applications and infrastructure) have made it easier to capture, manage and analyze data so as to piece together a more complete picture of customer behavior and of enterprise operations.

Case in point, an article from Business Week titled, “Math Will Rock Your World”, highlights companies such as Google, Aetna, Harrah’s and others that are using mathematics via analytical applications to sort out “swelling oceans of data” and mine data for insights to better understand customers.

While arguably a bit dated, the Business Week article showcases how companies are using customer data to build profiles and formulate models of both customers and employees that they believe will allow them to simulate and predict how to, “sell us things, steer us clear of diseases, and ramp up (employee) productivity.”

Other examples in the article show how companies are using advanced algorithms to make sense of unstructured data (emails, documents, call center notes), and optimize online advertising campaigns through the refinement and selection of keywords for search.

Using mathematics to better understand customers is serious business—just ask Netflix.

A Wired magazine article discusses how this online movie rental company offered a $1 million dollar prize to any one person or team that can improve its movie recommendation algorithm by 10%. In fact, according to Netflix’s leaderboard, a team from Bellkor just won the prize!

By opening access to one of the largest data sets available of online behavior—100 million customer movie ratings—Netflix has “crowdsourced” improvements to Cinematch, its engine that essentially recommends, “If you liked this movie, you’ll also like this one.”

However, even though it has taken over two years for a team to finally win the Netflix prize, minor improvements along the way to Cinematch have helped Netflix utilize more of its DVD inventory and improved customer loyalty as subscribers find movies of interest that perhaps they might have previously overlooked.

In another article, “Guessing the Online Customer’s Next Want”, Barney’s New York is mentioned as a company that’s seen dramatic marketing ROI improvement from using sophisticated analytical applications based on complex mathematics.

Through the use of technology, Barney’s is able to collect and analyze the online behavior of its customers and then craft smarter and more appropriate responses to interested audiences.

For example, the article notes, “An e-mail message announcing sales might go to those Web site visitors who had purchased certain products or types of products in the past, but who had done so only when the items were on sale. In the simplest terms, if someone buys only when something is on sale, but never buys anything in December, then the e-mail sale flier might not be sent to that customer in December.”

Just as in the early 1980s, when the financial industry was upended by the flight of quants from academia to Wall Street, marketers are starting to reap the brainpower of mathematicians, physicists and others as they codify their expertise and knowledge into sophisticated information technology systems and analytical applications. These innovative systems are helping marketers leverage information to better connect with customers and drive the business forward.

Paradigm shift? Absolutely. The world of marketing will likely never be the same again.

• Are you seeing these trends in your particular industry? If so, how so?
• The companies mentioned above are starting to treat data as one of their most valuable assets. Is your company on that path?
• Are you concerned with the potential “dark side” of simulating and modeling customer behavior—i.e. privacy issues?
• What skill sets will marketers need in the future to be able to compete in this new world of mathematics and marketing?

I’d love to hear from you!

Advanced Speech Technologies – Peril or Promise?

HAL 9000Enterprises are starting to deploy advanced speech technologies that can identify when a customer is angry, confused or even lying. By listening to call center feeds, these applications are often able to troubleshoot a given situation or route the call to a live agent with a specialization in solving critical problems.

But this nascent technology doesn’t always predict correctly—potentially causing even greater customer frustration. Are advanced speech technologies more peril than promise?

read original column

Gem of the Day: When Forecasting Pay Attention to Outliers

gemWhen forecasting in marketing, pay attention to dissent, especially when common knowledge or popular opinion says otherwise. More than a few CEOs and economists predicted the credit bubble of 2008 bursting in a big way and were mocked.

When you see opposing positions openly ridiculed, it’s time to consider the possibilities of those positions. If we believe that marketing has a role and responsibility in helping our companies develop sound, strong and robust business strategies, then we owe it to ourselves and our companies to pay attention to outliers.

Is Inventory Really Evil?

Inventory“Inventory is bad, inventory is evil,” finance and operations professors intone across business schools worldwide. And every B-school graduate knows companies should balance enough inventory to meet customer needs while accommodating shifting preferences. That said, companies face a paradox; holding too much inventory ties up valuable cash, but too little inventory is risky since some suppliers could lose their financial footing. In a global financial crisis, is inventory still evil?

Forecasting sales and inventory levels is probably one of the most difficult jobs of a product and/or supply chain manager as companies need to marry demand signals with supply. Adding more complexity to the mix is global supply chains that span weeks, multiple countries and sometimes oceans. Lots of hand-offs, tons of data to track, and lots of points for things to go wrong.

read more

One Step to Better Predicting the Future

crystal ballMarketers of all stripes are often tasked with forecasting—sales for next quarter or year, inventory levels to meet demand, or marketing budget to meet corporate goals.

However, the process of forecasting is often rife with bias, data quality issues, mathematical error, and/or poor planning assumptions. While no forecasting technique is perfect, predictions can be drastically improved through a simple technique: pulling your anchor.

read more