Extreme Redundancy – Don’t Leave Home Without It!

black swan

Visa’s command and control center—somewhere on the east coast of the United States—is a case study in redundancy with intense security, backups (for everything), and failover processes for system failures. And while most businesses don’t require these same levels of high availability for their own information networks and data centers, Visa can certainly teach a course on risk management to those who don’t believe such redundancy is necessary.

In the past four years, the unthinkable is happening more often than predicted. From the global stock market meltdown of 2008, to the Japanese tsunami disaster at Fukushima, to flooding in Thailand, and many other prominent examples, there’s Black Swans aplenty without signs of abatement.

Most of these events were near impossible to predict, but it’s entirely within the realm of risk management to prepare for heavy tails. The key to countering effects from extreme outlier events, says Black Swan author Nassim Taleb, is to build redundancy into business processes. He counsels companies, enterprises and even countries to “avoid optimization (and) learn to love redundancy.”

And while it may be a progressive example, Visa provides a well documented case study into managing fourth quadrant risk.  A Fast Company article titled, “Visa is Ready for Anything” provides a rare view in Visa’s data centers where 150 million daily transactions are processed.  Taking a page from NASA’s Mission Control, this data center monitors Visa network security, availability and capacity on a 24x7x365 basis. And meeting the definition of a “Tier 4” data center; “every major system—mainframes, air conditioners batteries etc, has a backup.”  With backups for backups and disaster planning in place, this data center is designed to withstand attacks from all sorts of events – terrorism, hackers, and even natural disasters such as earthquake and/or tornado.

While it’s true that Black Swans are about “unknown, unknowns”, and preparation for every type of risk is not only unrealistic but also too expensive, there is much companies can do to build more redundancy into daily operations. Redundancy is especially relevant as “interlocking fragility” of global communication, supply chain, and financial networks means correlations move to one with increasing frequency.

Today’s business culture celebrates “optimization” for every process; in essence living as near the edge as possible without falling off the cliff.  That’s a dangerous strategy for today’s interconnected world, especially because extreme events can take networks offline for hours, days and even weeks, costing companies tens of millions in lost revenues.

Critics of redundancy argue that robustness is too expensive. But Dr. Taleb counters; “Redundancy is like (being) long on an option, you certainly pay for it but it may be necessary for survival.”

Black Swan Alert: Low Tech Links Devastate High Tech Supply Chains


Find your supply chain best practices checklist. Just-in-time processes? Check.  Optimum sourcing strategies? Check. Lean logistics? Check. Best-in-class technologies? Check.  Black swan avoidance? Uh-oh.

Bangkok, Thailand is under water.  In perhaps the world’s worst flooding to devastate such a large city in modern times, Bangkok is home to ten million citizens and has an additional ten million people outside city limits.  The Chao Praya river drives down the middle of Bangkok and is supported by dikes and drainage canals for rainy seasons. But this month, the river is eight feet above normal, and while dikes have not failed, flooding has caused most residents to leave the city.

The human tragedy of such a flood aside, what’s occurring in Thailand is on just about every global CEO’s radar screen. And that’s because Bangkok is a crucial link in many high tech supply chains.

David Pilling of the Financial Times notes that companies such as Mazda, Toyota and Toshiba all have suppliers and factories in Thailand. And Honda’s Thai assembly plant, which traditionally manufactures 250,000 cars a year, has been shut down for three weeks.   With such interconnected and complex supply chains, sometimes spanning many countries, Pilling says these links are “prone to strain, particularly when paired with the just-in-time practices pioneered by Japan.”

Many supply chain gurus push companies to optimize inventory practices, adopt just-in-time strategies, and reduce redundancy wherever possible. And in “normal times” this approach makes sense. However, when disaster strikes, manufacturers are discovering their thin margins for error are in fact, leaner than they should be.

Nassim Taleb, author of The Black Swan, and professor of Risk Engineering at New York University is no stranger to interlocking fragility. He says most business professionals assume we live in a world of mild randomness, where events don’t stray far from the mean. Taleb argues the opposite, that in fact, because of tightly coupled financial and consumer markets (much less recurring natural disasters) we live in periods of wild randomness where small probability events carry large impacts (i.e. Black Swans).  These extreme events—Thai flooding or Fukushima disaster, for example, are shutting down entire supply chains for weeks and even months, causing hundreds of millions in lost profit opportunities.

A solution proposed by Dr. Taleb and others is to build a robust system (in this case supply chain) with redundancies and disaster recovery processes to properly manage extreme event risk. And while some experts argue redundancy adds costs to already paper thin margins, there are surely costs to not supplying marketswith needed product because an unforeseen event has silenced your supply chain.

Bob Lutz, former vice chairman of General Motors says it best; “Running your procurement purely on a short term, point in time, cost minimization model is like shopping for rock bottom home insurance. It looks real smart until your house burns down.”  He goes on to say; “What happens when ‘just in time’ is ‘just plain late’”?

Good question and one which many global CEOs are still struggling to answer.

USAF Prevents Insomnia with Geospatial


What causes sleepless nights? For sure, too much caffeine can keep your eyelids open, however it’s more likely that worry and anxiety are the source. Fortunately, insomnia’s not the norm for United States Air Force (USAF) general officers and other government officials. That’s because with geospatial capabilities the USAF knows where every aircraft, piece of equipment and part is located and where it’s been—anywhere in the world.

Image Courtesy of Google Maps

Geospatial (also known as location intelligence) is a technology that allows enterprises to store, process and consume geographically based data. Geospatial data can be points (longitude, latitude), lines (a series of points), or even polygons representing a defined boundary. With geospatial, an enterprise can measure distances between two objects, or identify objects within, touching or traversing a specified perimeter. And while approximately 80% of corporate data has a location reference, there are plenty of enterprises not taking advantage of this valuable location information.

At Teradata Partners conference 2011 in San Diego, California, speakers Tim Cotton and Teal Walker presented how geospatial capabilities benefit the USAF. Specifically, leveraging data integrated from over one-hundred sources into Teradata database and Google Maps for visualization purposes, the USAF provides its users trusted information for inventory control including drill down capabilities to the part and supply level. And with geospatial, officials can visually examine where a particular asset has been throughout its lifecycle.

Teradata and Google Maps also help the USAF visually monitor “exceptions” via alerts—in real time—to track movement of materials, vehicles, commodities and assets. Now any aircraft or equipment going to the wrong place can be identified quickly and rerouted.  And lastly with geospatial capabilities, the USAF can easily perform “proximity analysis” to immediately discover assets nearby and available to maintain mission support.

Geospatial enables “big picture” location analysis. The USAF knows where its assets are located at all times, and can direct them quickly should the need arise. Important operational details are captured and readily available via this geospatial solution, allowing government officials to clear their minds and get a good night’s sleep.

The USAF can pinpoint and direct its most productive assets at a moment’s notice. Shouldn’t your business have this same advantage?

Black Swans Causing a Rethink on Global Supply Chains?

A fair percentage of academicians and corporate supply chain experts agree that running supply chains “lean and mean” with just in time processes is the optimum approach. However, Black Swans (low probability events with high impact) such as the March 11, 2011 Sendai earthquake and subsequent tsunami have companies rethinking how they source to deliver the products and services critical to success.

The 8.9 magnitude earthquake and tsunami that ravaged Japan’s coast and cascading Fukushima nuclear calamity have severely tested the links in today’s global supply chains. And while it’s true that some Japanese electronics and automobile manufacturers continue to struggle with factory shutdowns, most global supply chains weathered the storm quite well. However as supply chains arguably become more complex, executives should reconsider how man-made and natural disasters can affect processes in delivering goods and services to market.

A Financial Times article titled “Industry Left High and Dry” cites how some companies were not prepared for the repercussions of Japanese factory slowdowns or shutdowns. These enterprises relied too much on Japanese factories to produce critical electronic components, and others simply lacked contingency disaster planning. Even more disturbing, Barry Tarnef, a risk manager at US insurance group Chubb says; “Many manufacturers (lacked) the information needed to mobilize alternative arrangements quickly (when their supply chains were interrupted).

One of the themes that I have highlighted over the past 2-3 years is with the global integration of trade, capital, information and labor over the past decade; few people understand that our world is more tightly connected than ever before. In a highly inter-dependent environment, small mistakes can have large ramifications. And even larger events – such as tsunami, volcanic ash and the like can cause an earnings miss or worse depending on the severity and criticality of the event.

That’s why companies are starting to re-consider their global supply chain processes. This means structuring alternative supply networks, adding regional suppliers (instead of relying solely on one country), and doing a better job of setting up an IT infrastructure capable of linking suppliers and providing supply chain visibility from “field to fork”.

Black Swan author Nassim Taleb has long counseled companies to consider concepts of robustness and redundancy to build flexibility in response to adverse events. However, these considerations surely add additional costs, and many companies from retailers to manufacturers are already relying on paper thin margins.

One thing is for certain, there are no excuses for ignorance of how your supply chain processes work – or don’t work. The supply chain that weathers today’s storm may be wholly inadequate for tomorrows.


* For supply chains, should companies continue to “go lean” or build in redundancy? What are your thoughts?

Supply Chain Traceability – What’s in Your T-Shirt?

Does it concern you that cotton from a t-shirt originated in Uzbekistan? You might care if reports are true that cotton harvested from this country came from forced or child labor. In fact, consumers are increasingly asking manufacturers and retailers to ensure goods and services are produced with sustainable and ethical business practices. With goods processed between countries and multiple supplier touches, traceability in the supply chain is more important than ever.

Consumers are becoming more and more interested in the make-up of products they’re buying. The ability to trace products through the entire supply chain—whether from field to fork or source to consumer—is becoming a competitive weapon for some companies, especially if they can authenticate then promote goods produced with sustainable business practices.

A Financial Times article cites that ethical spending has increased significantly in the past 20 years. As an example in the United Kingdom, “The amount of spending and investment influenced by ethical considerations almost doubled between 1999 and 2008 to reach £36 billion.” The FT article mentions the growth of fair trade and certified organic products as part of this growth trend. And fellow DailyFix author, Ted Mininni, says these types of goods, “are increasingly being added to retail assortments”—and in growing numbers!

However, one of the major challenges for both retailers and manufacturers alike is authenticating ethically sourced products. There’s more to the process than taking the supplier’s word for it.

In the case of organic bananas, the FT article notes, “the food company Dole labels each of its organic bananas with a three-digit number that, when entered on its website, reveals details of the farm where that banana is grown.” Identifying where a banana is sourced, however, is simple compared to such products as sweaters or t-shirts that pass through multiple suppliers and countries.

In “The Elephant and the Dragon,” author Robyn Meredith confirms that products often “zigzag” through the global supply chain from factory to factory. “A cheap toy may be assembled by parts from 12 different factories,” she says. And something as sophisticated as an automobile might contain five to seven thousand parts.

Fortunately for marketers, the challenge is not beyond the capabilities of today’s technologies. Some progressive companies are engaging in a purposeful effort to build effective policies (including auditing), technologies (supply chain analytics and infrastructure) and processes to track and monitor the extended supply chain.

Of course, implementing a data-driven supply chain infrastructure is only half the battle—supply chain managers, operations personnel and marketers must learn how to use it! Marketers, working alongside operations, will need training on the various tools and systems used to access data for reporting and query purposes. And marketers may also choose to make supply chain data available directly to consumers—similar to Dole’s web portal—thus enabling them to verify product origins for themselves!

Ethical sourcing, trade, manufacturing and retailing will continue to be a hot button for consumers. However, as seen from this article, jumping into this marketplace requires much more than just fancy signage and/or promotion. A real commitment to corporate responsibility and sustainable practices must be much more than lip service; it involves significant investment in people, processes, technology, and strategy. As seen from the complexity in supply chain traceability alone, it’s definitely not an effort a company should take lightly.


• Does it matter to you how a product is made? Are you interested in the origins of the products and services you consume?

• The Financial Times article says that companies should make information about their supply chain public—or consumers will do it for them. Do you agree with this statement?

Why Capacity Management Matters to Business Executives

bucketIn a challenging global slowdown, the world seems awash in capacity. Scans of major business publications show airlines reducing flights, companies furloughing or firing employees, and manufacturers closing plants. If you agree that it appears there is more unused capacity than demand, why should capacity management matter?

It would seem in the “Great Recession,” capacity planning and management should be a minimal consideration. In fact, capacity utilization for many industries is at an all time low.

For example, from January 2008 to January 2009, according to the Financial Times, the demand for automobiles in the United States fell from 15.9 million to 9.6 million per year. And Wall Street Journal reports Federal Reserve Chairman Ben Bernanke told the House Budget Committee recently, “The slack in resource utilization remains sizable”.

As companies attempt to cope with a “new normal,” painful restructuring processes have included reducing “capacity” in human resources, plants and equipment, information technology, number of brands, distribution channels, and even debt covenants. All this restructuring is intended to pare down capabilities to what is perceived as a new reality in market conditions.

Indeed, observing macro-economic conditions, it’s tempting to write off “growth.” However, “growth” is far from dead.

Take for example, the exponential growth trends of Facebook and Twitter. In January 2009, Facebook touted its 150 millionth user, and in May 2009 surpassed 225 million users! One site projects Facebook to have 300 million users by the end of the year. Twitter’s growth has also been phenomenal—audiences grew 40% in just 30 days (March-April 2009).

In fact, “growth” exists (often exponentially) in areas such as data volumes, populations, energy usage, Moore’s Law, GDPs of select countries (India, China etc), education expenditures, and unfortunately—state and national debts!

Growth also can be found in micro-segments and categories such as increases in market share of private label brands vs. national brands on grocery store shelves, or Apple’s share of the smartphone market. Once our eyes are opened to growth trends, it’s quite easy to see signs of expansion everywhere!

The ability to meet the needs of your customers now and in the future is a critical function of any business. That’s what capacity management is all about. Spikes in demand could mean that your company is leaving money on the table and/or failing to meet customer needs. Need proof? For customer reaction, simply perform a web search on keywords “Twitter down time” or “Twitter outage” and you’ll gain evidence of how important capacity management really is.

Indeed, capacity management isn’t a one-time, annual event. It should be a continual process of making sure your business can scale up or down to meet customer needs. With a thumb on the pulse of demand, marketers have a responsibility to help establish a well documented capacity plan and process that considers future business requests.

Sound like simple, common sense, right? Properly predicting demand is anything but easy. Considerations must include a clean and accurate set of historical data, an analytical infrastructure to compute and analyze data, an understanding of the current state of the business and its capabilities, future growth projections based on applicable trends, and then a gap analysis of what it would take to scale based on various “what-if” scenarios.

Capacity management is all about reducing surprises. Take a good, hard look at your business. What’s growing? Something surely is.

What marketing campaigns are you preparing? What happens—for goodness sake—if they’re too successful and demand exceeds available supply? McDonald’s in India had to scale back marketing campaigns for Chicken McNuggets because they couldn’t keep up with demand. Good marketing is making promises your company CAN deliver.

Can you accurately predict if and when you’ll run out of resources to meet customer needs? Can you afford not to properly manage “capacity”?


• There appears to be a glut of capacity worldwide (i.e. shipping, telecommunications, manufacturing etc.). Should marketers be concerned with the concept of capacity management?
• What are the ramifications of getting capacity management wrong?
• Businesses are adding flexibility to meet spikes in demand through vehicles like cloud computing, temporary labor and outsourcing. Can you think of others?
• Suppose “capacity management” is built into the function of an annual strategic planning exercise. What might be a pitfall of this approach?

Marketers – What’s in Your Supply Chain?

ostrich, head in sandWith global sourcing strategies in place, companies often assemble finished goods from raw materials from hundreds of suppliers.

However, not all suppliers act ethically, and some take short-cuts in quality control. In order to properly manage our brands and take ownership of the “customer experience”—marketers need visibility into the supply chain.

 Do you know what’s in your supply chain? read original post