When Ideology Reigns Over Data

Increasingly, the mantra of “let the data speak for themselves” is falling by the wayside and ideology promotion is zooming down the fast lane. There are dangers to reputations, companies and global economies when researchers and/or statisticians either see what they want to see—despite the data, or worse, gently massage data to get “the right results.”

Courtesy of Flickr. By Windell Oskay
Courtesy of Flickr. By Windell Oskay

Economist Thomas Piketty is in the news. After publishing his treatise “Capital in the Twenty First Century”, Mr. Piketty was lauded by world leaders, fellow economists, and political commentators for bringing data and analysis to the perceived problem of growing income inequality.

In his book, Mr. Piketty posits that while wealth and income were grossly unequally distributed through the industrial revolution era, the advent of World Wars I and II changed the wealth dynamic as tax raises helped pay for war recovery and social safety nets. Then, after the early 1970s, Piketty claims that once again his data show the top 1-10% of earners take more than their fair share. In Capital, Piketty’s prescriptions to remedy wealth inequality include an annual tax on capital and harsh taxation of up to80% for the highest earners.

In this age of sharing and transparency, Mr. Piketty received acclaim for publishing his data sets and Excel spreadsheets for the entire world to see. However, this bold move could also prove to be his downfall.

The Financial Times, in a series of recent articles, claims that Piketty’s data and Excel spreadsheets don’t exactly line up with his conclusions. “The FT found mistakes and unexplained entries in his spreadsheet,” the paper reports. The articles also mention that a host of “transcription errors”, “incorrect formulas” and “cherry-picked” data mar an otherwise serious body of work.

Once all the above errors are corrected, the FT concludes; “There is little evidence in Professor Piketty’s original sources to bear out the thesis that an increasing share of total wealth is held by the richest few.” In other words, ouch!

Here’s part of the problem; while income data are somewhat hard to piece together, wealth data for the past 100 years is even harder to find because of data quality and collection issues. As such, the data are bound to be of dubious quality and/or incomplete. In addition, it appears that Piketty could have used some friends to check and double check his spreadsheet calculations to save him the Ken Rogoff/Carmen Reinhardt treatment.

In working with data, errors come with the territory and hopefully they are minimal. There is a more serious issue for any data worker however; seeing what you want to see, even if the evidence says otherwise.

For example, Nicolas Baverez, a French economist raised issues with Piketty’s data collection approach and “biased interpretation” of those data long before the FT report.  Furthermore, Baverez thinks that Piketty had a conclusion in mind before he analyzed the data. In the magazine Le Point, Baverez writes; “Thomas Piketty has chosen to place himself under the shadow of (Karl Marx), placing unlimited accumulation of capital in the center of his thinking”.

The point of this particular article is not to knock down Mr. Piketty, nor his lengthy and researched tome. Indeed we should not be so dismissive of Mr. Piketty’s larger message that there appears to be an increasing gap between haves and have nots, especially in terms of exorbitant CEO pay, stagnant middle class wages, and reduced safety net for the poorest Western citizens.

But Piketty appeared to have a solution in mind before he found a problem. He will readily admit; “I am in favor of wealth taxation.”  When ideology drives any data driven approach, it becomes just a little easier to discard data, observations and evidence that don’t exactly line up with what you’re trying to prove.

In 1977, statistician John W. Tukey said; “The greatest value of a picture is when it forces us to notice what we never expected to see.” Good science is the search for causes and explanations, sans any dogma, and willingness to accept outcomes contrary to our initial hypothesis. If we want true knowledge discovery, there can be no other way.

 

Excel Model Errors – Don’t Throw the Baby out with the Bathwater

Two noted economists, Kenneth Rogoff and Carmen Reinhardt, recently had their findings on country debt to GDP ratios questioned, as it was discovered an Excel spreadsheet error led to some grave miscalculations. And while plenty of financial bloggers and economists took the opportunity to gloat over Rogoff and Reinhardt’s misfortune, there is a larger point here: just because mathematical calculations are wrong, it doesn’t mean a particular idea isn’t directionally sound.

Courtesy of Flickr. By BlackLineSystems
Courtesy of Flickr. By BlackLineSystems

In 2010, Rogoff and Reinhardt published a paper on the link between high public debt and slower economic growth. Their findings showed that when a country reached a debt level of greater than 90% of GDP, that country’s growth would slow to a crawl. This paper was subsequently used as the empirical basis for fiscal austerity—or belt tightening—for many European countries.

However, since the publishing of Rogoff and Reinhardt’s 2010 paper, their findings have been under intense scrutiny. Facing pressure to release their methodology and data, Rogoff and Reinhardt finally let other statisticians examine the study’s underlying calculations.

When Rogoff and Reinhardt’s Excel spreadsheets were released, a pair of graduate students discovered some coding errors. One key error omitted five countries from the calculations, which changed the mean of negative 0.1% economic growth to a positive 2.2%, a pretty significant switch! In other words, the conclusion that the “magic number of 90% debt to GDP equates to slow growth” wasn’t so magical after all.

Predictably, mainstream economists like Paul Krugman were quick to pounce. In his column, “Holy Coding Error, Batman”, Krugman called the error “embarrassing”, a “failure” and concluded it was reason enough to discount the underlying message that countries with higher debt could see slower growth in the future.

Krugman’s gloating aside, we should note that just because calculations supporting a particular idea are wrong, it doesn’t necessarily mean the proverbial “baby” should be tossed out with the “bathwater”.

Here’s why: an article on WSJ’s Market Watch cites a few studies showing 88% of spreadsheets contain errors of some kind. Ray Panko, a professor of IT management at University of Hawaii says that spreadsheet “errors are pandemic”.

Now whether you believe the 88% number is correct, or even if you discount it by half—as a consultant friend of mine suggests—it’s still a whopper of a number!

Going forward, with the knowledge a fair percentage of excel calculations are likely flawed in some manner, it makes sense that while we should expect the numbers supporting an idea need to be accurate, we should also understand that there could be errors. And because there could be calculation errors, we need to decide if the idea—outside any erroneous calculations—is a sound idea, or not.

Of course, there are instances where it’s critical to get mathematical calculations correct such as launching rockets, landing planes, engineering a building or bridge etc. But let’s also be careful not to immediately throw away an idea as “false” simply because it’s discovered someone made a correctible excel spreadsheet error.

Getting back to the Rogoff and Reinhardt commotion, this is exactly what Financial Times columnist Anders Aslund has in mind when he writes, “(While) the critique of Reinhart and Rogoff correctly identifies some technical errors in their work, one cannot read it and conclude the case for austerity is much weakened. High public debt is still a serious problem.”  I would add this is especially true for countries where their debt is not denominated in their own currency.

With the realization that most spreadsheets have errors, we should check, double check, and triple check Excel calculations to ensure accuracy. Peer review of excel calculations is also a recommended approach.

But let’s also not be so quick to throw out perfectly good ideas where it’s discovered some excel miscalculations, or omissions have skewed the results.  After all, a key idea may not be precisely supported by the maths, but still may be directionally correct.  Or as New York Fund Manager Daniel Shuchman says, we don’t need to touch the stove to prove it’s hot.

Questions:

What are the key lessons in the Rogoff and Reinhardt debacle?  Mistakes in treating correlation for causation? Sloppy coding? Applying too much historical data where conditions may have changed?  Applying too little data (cherry-picking)? What say you?

Analytics and Hedgehogs: Lessons from the Tampa Bay Rays

The Tampa Bay Rays spend significantly less on payroll than some of the wealthier teams in Major League Baseball, but get results that are sometimes better than those that wildly overspend. The Tampa Bay Rays success boils down to two things – understanding how to be a hedgehog, and continual application of statistics and analytics into daily processes.

Tampa Bay RaysGreek poet Archilochus once said: “The fox knows many things, but the hedgehog knows one big thing.” Many interpretations of this phrase exist, but one characterization is the singular focus on a particular discipline, practice or vision.

According to a Sports Illustrated article “The Rays Way”, while Major League Baseball teams such as the Los Angeles Angels load up on heavy hitters such as Albert Pujols and Josh Hamilton, the Tampa Bay Rays instead have a hedgehog-like and almost maniacal spotlight on pitching.

For example, SI writer Tom Verducci says “The Rays are to pitching what Google is to algorithms.” In essence, the Rays have codified methods (on how to raise up young pitchers and injury prevention techniques) and daily processes (including exclusive stretching and strengthening routines) into a holistic philosophy of “pitching first”.

But enabling that hedgehog-like approach to pitching is a culture of measurement and analysis.  To illustrate, the SI article mentions that pitchers are encouraged to have a faster delivery (no more than 1.3 seconds should elapse between a pitch and hitting the catcher’s glove). Pitchers are also instructed to throw the changeup on 15% of deliveries. And while other pitchers try and focus on getting ahead of batters, the Rays have discovered it’s the first three pitches that matter, with the third being the most important.

In terms of applying analytics, the Rays rely on a small staff of “Moneyball” statistical mavens that provide pitchers with a daily dossier of the hitters they’ll likely face, including they pitches they like and those they hate. And analytics also plays a part in how the Rays position their outfield and infielders to field balls that might otherwise go into the books as hits.

The Rays are guarded about sharing their proprietary knowledge on processes and measurement, and for good reason, as last year they had the lowest earned run average (ERA) in the American League and held batters to the lowest batting average (.228) in forty years. Even better, they’ve done this while spending ~70% less than other big market teams and winning 90+ games three years in a row. That’s nailing the hedgehog concept perfectly!

Seeing a case study like this, where a team or organization spends significantly less than competitors and gets better results, can be pretty exciting. However, an element of caution is necessary. It’s not enough to simply follow the hedgehog principle.

The strategy of a hedgehog-like “focus” can be highly beneficial, but in the case of the Tampa Bay Rays, it’s the singular focus on a critical aspect of baseball (i.e. pitching), joined with analytical processes, skilled people and the right technologies that really produce the winning combination.

How They Fit Together: Bell Curves, Bayes and Black Swans

Probability is defined as the possibility, chance or odds of likelihood that a certain event or occurrence will take place now or in the future.  In a world where business managers like to “know the odds”, how does probabilistic thinking (Frequentism and Bayesian) mesh with extreme events (i.e. Black Swans) that just cannot be predicted?

Image Courtesy of Wikipedia

Statisticians lament how few business managers think probabilistically. In a world awash with data, statisticians claim there are few reasons to not have a decent amount of objective data for decision making. However, there are some events for which there are no data (they haven’t occurred yet), and there are other events that could happen outside the scope of what we think is possible.

The best quote to sum up this framework for decision making comes from the former US Defense secretary Donald Rumsfeld in February 2002:

“There are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns – there are things we do not know we don’t know.”

Breaking this statement down, it appears Mr. Rumsfeld is speaking about Frequentism, subjective probability (Bayes) and those rare but extreme events coined by Nassim Taleb as “Black Swans”.

Author Sharon Bertsch McGrayne elucidates the first two types of probabilistic reasoning in her book “The Theory That Would Not Die”.  Frequentism (conventional statistics), she says, relies on measuring the relative frequency of an event that can be repeated time and again under the same conditions. This is the world of p-values, bell curves, coin flips, casinos and actuaries where data driven decision making is objective based on sampling or computations of large data sets.

The greater part of McGrayne’s tome concentrates on defining Bayesian Inference, or subjective probability also known as a “measure of belief”. Bayes, she says, allows making of predications with no prior information at all (no frequency of events).With Bayes, one makes an educated guess, and then keeps refining that guess based on new information, thus updating and revising the probabilities, and getting “closer to certitude.”

Getting back to Rumsfeld’s quote, Rumsfeld seems to be saying we can guess the probability of  “known knowns” because they’ve happened before and we have frequent data to support objective reasoning. These “known knowns” are Nassim Taleb’s White Swans. There are also “known unknowns” or things that have never happened before, but have entered our imaginations as possible events (Taleb’s Grey Swans). We still need probability to discern “the odds” of that event (e.g. dirty nuclear bomb in Los Angeles), so Bayes is helpful because we can infer subjective probabilities or “the possible value of unknowns” from similar situations tangential to our own predicament.

Lastly, there are “unknown unknowns”, or things we haven’t even dreamed about (Taleb’s Black Swan).  Dr. Nassim Nicholas Taleb labels this “the fourth quadrant” where probability theory has no answers.  What’s an illustration of an “unknown unknown”? Dr. Taleb gives us an example of the invention of the wheel, because no one had even though or dreamed of a wheel until it was actually invented. The “unknown unknown” is unpredictable, because—like the wheel—had it been conceived by someone, it would have been already invented.

Rumsfeld’s quote gives business managers a framework for thinking probabilistically. There are “known knowns” for which Frequentism works best, “unknown knowns” for which Bayesian Inference is the best fit, and there is a realm of “unknown unknowns” where statistics falls short, where there can be no predictions. This area outside the boundary of statistics is the most dangerous area, says Dr. Taleb, because extreme events in this sector usually carry large impacts.

This column has been an attempt to provide a decision making framework for how Frequentism, Bayes and Black Swans fit together—by using Donald Rumsfeld’s quote.

What say you, can you improve upon this framework?

“Moneyball” Takes the Next Big Leap

Spying inefficiencies – in markets, business models, business processes such as supply chains and more has been one of the key methods companies gain competitive advantage.

Take for example an ecosystem as staid as Major League Baseball (MLB) which counts the year 1869 as its inception. Michael Lewis’ Moneyball tome showcases how Oakland A’s general manager Billy Beane revolutionized how baseball talent is scouted, sourced and even utilized on the field. Beane knew that in Oakland that he’d never have the payroll to compete with big market teams such as the Boston Red Sox or New York Yankees.  So Beane had to find a new method of competing.

By now, those who have read Moneyball or at least seen the Hollywood movie know Beane chose to compete on analytics. He summoned help from statisticians and computer programmers to define new metrics, and resource talent that he felt was either undervalued or overlooked. And for three to four years, the Oakland A’s competed for American League pennants with scrappy efficiency—and usually with half the payroll of large competitors.

Competitive advantage, however, doesn’t usually last long. Indeed, other MLB teams learned of Beane’s methods (and devoured any Bill James publication they could find) to neutralize Beane’s gains in the marketplace.  Equilibrium, or parity, was once again reached.

That is, until Moneyball concepts of “competing on analytics” took the next leap from helping analyze and assemble teams to improving performance of individual players.

In MLB it’s common for players to not meet their so-called potential. For example, Billy Beane himself was a fine baseball player with all the requisite skills necessary to achieve stardom. However, Beane never quite lived up to expectations of teams that purchased his skills—he left MLB with a .219 career hitting average and only three home runs. Exhibit B is current Oakland A’s pitcher Brandon McCarthy, who despite much promise, struggled for years with his command and control—relying on only two good pitches to get hitters out.

That is until McCarthy got the Moneyball religion. An ESPN Magazine article titled “SaviorMetrics” chronicles McCarthy’s struggles on the mound until 2009 when he adopted Moneyball principles. Scouring websites devoted to advanced MLB statistics, McCarthy discovered he was one of the worst pitchers in getting hitters to bat ground balls. That’s especially significant when one realizes, according to the ESPN article that, “the average flyball is three times more dangerous than the average ground ball” in producing runs.

McCarthy decided to retool his approach and adopt two new pitches that were sure to induce ground balls. It wasn’t magic at first, but over a few years of trial and error, McCarthy found his groundball to flyball (GB/FB) ratio doubled—which of course meant fewer of his pitches would end up in the outfield bleachers.

Of course, there were mechanical issues to iron out along the way, plus McCarthy had to adopt two new pitches (which certainly wasn’t easy).  But so far, his strategy of inducing more ground balls seems to be working–he just signed a one year contract with the Oakland A’s for more than $4 million.

Inefficiencies are everywhere and they can be exploited for competitive gain. Sometimes those inefficiencies are in markets (exploited through trading arbitrage strategies), in supply chains, in ecosystems (i.e. Major League Baseball), or even for individual performances (i.e. Brandon McCarthy).

The trick is to first detect those inefficiencies with data analysis, create a plan to attack/exploit them for gain, and to execute on that plan. And of course, it’s important to evolve and always seek new avenues of improvement.

In Brandon McCarthy’s case, it’s only a matter of time before hitters adapt to his strategies and learn his new pitches. Here’s hoping he has another analytics discovery up his sleeve.

It Takes Courage to Compete on Analytics

Billy Beane’s “Moneyball” approach to developing and staffing a professional baseball team has come under intense scrutiny as long time major league scouts and analysts take delight in sub-par performances of the Oakland Athletics.  Beane however seems undaunted in using statistical analysis to undercover market place inefficiencies.  Indeed, it takes courage to solve problems in a whole new—perhaps heretical—manner. And while plenty of folks take pleasure in Beane’s recent comeuppance, there’s a good chance he’s already had the last laugh.

By now, most business professionals have either read Michael Lewis’ “Moneyball” or at the very least seen Brad Pitt’s rendition of Billy Beane in Hollywood’s version.  Beane’s claim to fame is that he had the courage to defy decades of “common knowledge” in assembling a winning baseball team.  For many years, winning teams were accustomed to paying top dollar for the best hitters and pitchers evidenced by metrics such as batting average and earned run average respectively.

However, in the small-town Oakland market, Beane didn’t have the luxury of affording a high payroll. And if he wanted to compete, he needed to find a different (and perhaps better) method of keeping up with the likes of the Boston Red Sox and New York Yankees.

Beane knew there were “inefficiencies” in baseball markets that could be exploited; he just had to discover those undervalued metrics.

Enlisting the help of his Harvard friend Paul Depodesta, Beane used statistical analysis to discover metrics that—in his opinion were more indicative of baseball success—such as on base percentage and slugging percentage for batters, or the ability to get “outs” for pitchers.  Then Beane went about acquiring a motley bunch of players that most teams discarded as near worthless, and positioned each player for success based on his analytical models.

And for a while it worked. With one of the league’s lowest payrolls the Oakland A’s competed with teams spending 2-3x in overall salaries. In fact, a Sports Illustrated article mentions; “From 2000 to 2006, (Oakland A’s) averaged 95 wins, captured four AL West titles and made five playoff appearances.”

Alas, all competitive advantage is fleeting. As Paul Depodesta left for greener pastures, and a few of Beane’s disciples took other league jobs, major league baseball teams figured out what made Moneyball work and effectively neutralized the Oakland A’s advantage.  With other teams adapting to Moneyball tactics, from 2007 and beyond, the Oakland A’s were a .500 team at best, and sometimes much worse.

And in the process of adopting Moneyball tactics, Billy Beane made plenty of enemies. The “old guard” definitely had a preferred way of defining baseball success and would often scoff at the “fat bodies” Beane employed, and laugh at the un-orthodox pitchers he would trot out in the ninth inning.  And in regard to the lackadaisical record of the Oakland A’s in recent years, one scout sneered; “So much for the genius … He doesn’t look so smart anymore, does he?”

Competing on analytics takes courage. In response to analyst and pundit criticism leveled at Beane for his Moneyball approach, author Michael Lewis says; “Beane had the nerve to seize upon ideas rejected, or at least not taken too seriously, by his fellow Club members, and put them into practice.”  For a while, Beane had the advantage. It might not have been pretty, but the Oakland A’s were effective at winning as many games as other prestigious teams, for half the cost.

And Beane ultimately may have the last laugh. Today, there are teams like the Boston Red Sox that take the best of Moneyball tactics and overlay them with powerful financial resources to take baseball competitiveness to a whole new level. Baseball has been changed forever from a “gut feel” business to one that’s analytically driven.

Fortunately for today’s analytical professionals, “competing on analytics” is taking hold as more companies understand they’re sitting on literal goldmines of structured and multi-structured data just begging for analysis. But analytics users be forewarned, there’s still plenty of “old guard” that will put up roadblocks, enlist subterfuge and even openly mock your new data driven approaches.

Michael Lewis notes that Beane didn’t invent sophisticated analytical analysis—he just had the courage to use it to create competitive advantage and shake up a traditionally stodgy industry.  There are plenty of industries ripe and ready for the business value that analytics will eventually unlock. Are you ready for the challenge?

The Human Factor Continually Confounds Probability Models

With four weeks to go in the 2011 Major League Baseball season, the probability of the Boston Red Sox of making the playoffs was 99.6%. And most of us know the story; in one of the biggest collapses in baseball history, the Red Sox tanked a nine game lead and served the wild card slot to the Tampa Bay Rays. In creating “one for the record books”, the 2011 Red Sox show us that the human factor continually confounds probability models.

Some things aren’t supposed to happen. The 2011 Boston Red Sox certainly should not have missed the playoffs with a nine game lead, and the 1995 Anaheim Angels should not have finished their year 12-26 (losing a nine game lead and missing the playoffs). Moreover, probability models said the stock market (DJIA) should not have lost 54% of its value in the 2008 “Great Recession”.

There’s definitely a danger in too much reliance on normal distribution probability models, especially when humans are concerned says Financial Times writer John Authers.

Studying the 2011 Boston Red Sox, Authers suggests the team may have been overconfident in statistics since few teams in baseball history had collapsed with such a lead.

Authers also believes bell curve probabilistic models would not have been a reliable indicator of possible failure because such models assume event independence where one event should not affect another. But those who follow sports understand the concept of “momentum in a game”, or even from game-to-game where a team can feed off past success to gain confidence.

In reference to the 2008 market crash, Steven Solmonson, head of Park Place Capital Ltd said; “Not in a million years would we have expected this gyration to be as vicious and enduring as it has been.”  And I’m sure that Boston Red Sox fans didn’t believe their team could lose a significant lead over the Tampa Bay Rays with just a few games left in the season.

Whenever humans are involved, the lesson is clear: don’t get over confident in normal distribution probability models. Next thing you know, you might get slapped (or worse) by the fat tail.

Follow

Get every new post delivered to your Inbox.

Join 41 other followers

%d bloggers like this: