NSA and the Future of Big Data

The National Security Agency of the United States (NSA) has seen the future of Big Data and it doesn’t look pretty.  With data volumes growing faster than the NSA can store, much less analyze, if the NSA with hundreds of millions of dollars to spend on analytics is challenged, it raises the question; “Is there any hope for your particular company”?

Courtesy of Flickr. By One Lost Penguin

By now, most IT industry analysts accept the term “Big Data” is much more than data volumes increasing at an exponential clip. There’s also velocity, or speeds at which data are created, ingested and analyzed. And of course, there’s variety in terms of multi-structured data types including web logs, text, social media, machine data and more.

But let’s get back to data volumes. A commonly referenced report conducted by IDC mentions data volumes are more than doubling every two years. Now that’s exponential growth that Professor Albert Bartlett can appreciate!

What are consequences of unwieldy data volumes? For starters, it’s nearly impossible to effectively deal with the flood.

In James Bamford’s “Shadow Factory”, he mentions how the NSA is vigorously constructing data centers in remote and not so remote locations to properly store the “flood of data” captured from foreign communications including video, voice, text and spreadsheets.  One NSA director is quoted as saying; “Some intelligence data sources grow at a rate of four petabytes per month now…and the rate of growth is increasing!”

Building data centers and storing petabytes of data isn’t the end goal. What the NSA really needs is analysis. And in this area the NSA is falling woefully short, but not for lack of trying.

That’s because in addition to the fastest super computers from Cray and Fujitsu, the NSA needs programmers who can modify algorithms on the fly to account for new key words that terrorists or other foreign nationals may be using. The NSA also constantly seeks linguists to help translate, document and analyze various foreign languages (something computers struggle with—especially discerning sentiment and context).

According to Bamford, the NSA sifts through petabytes of data on a daily basis and yet the flood of data continues unabated.

In summary, for the NSA it appears there are more data to be stored and analyzed than budget to procure more supercomputers, programmers and analytic talent.  There’s just too much data and too little “intelligence” to let directors know what patterns, links and relationships are most important. One NSA director says; “We’ve been into the future and we’ve seen the problems of a “tidal wave” of data.”

So if one of the most powerful government agencies in the world is struggling with an exponential flood of big data, is there hope for your company?  For advice, we turn to Bill Franks, Chief Analytics Officer for Teradata.

In a Smart Data Collective article, Mr. Franks says that even though the challenge of Big Data may be initially overwhelming, it pays to eat an elephant a single bite at a time. “People need step back, push the hype from their minds, and think things through,” he says.  In other words, don’t stress about going big from day one.

Instead, Franks counsels companies to “start small with big data.”  Capture a bit at a time, gain value from your analysis and then collect more he says. There’s an overwhelming temptation to splurge on hundreds of machines and lots of software to capture and analyze everything. Avoid this route, and instead take the road less traveled—the incremental approach.

The NSA may be drowning in information, but there’s no need to inflict sleepless nights on your IT staff.  Think big data but start small. Goodness knows, in terms of data, there will always be plenty more to capture and analyze. The data flood will continue. And from a IT job security perspective, that’s a comforting thought.

Ring in the New Year with New Data Products

For web-based businesses, and of course, those with a web presence (which is just about everyone) there’s a goldmine of behavioral data accessible with the right tools. The trick is getting past static web analytic reporting (bounce rates, page views, session times etc.) and going further into unlocking the rich treasure trove of machine data, text and weblogs that create “big data” insight.

In your business, gigabytes if not terabytes of multi-structured data are likely just waiting to be coupled with your imaginative thinking and analysis to create new data products that ultimately help drive customer interactions and revenues

For example, take a look at what LinkedIn is doing in creating new “products” with data they collect and analyze with a MapReduce approach and other techniques.

According to a recent whitepaper “Building Data Science Teams”, LinkedIn’s former Chief Data Scientist shows how smart thinking can be paired with compute power and huge quantities of multi-structured data to create innovative new products such as:

  • Products that provide personalized content (which makes customers feel products/services are handpicked for them based on their wants/needs)
  • Products that drive the company’s value proposition (For LinkedIn, it’s their “People You May Know” or “Jobs You May Be Interested In” algorithms which drive further customer engagement)
  • Products that facilitate an introduction to other products (to funnel customers into other relevant areas of your website and thus lower your bounce rates)
  • Products that prevent dead ends (ex: smart algorithms that suggest other potential purchases, i.e. “People like you also bought…”)

And of course, many of the above “products” are more than simply focused on nebulous metrics such “customer engagement”—they can directly tie to revenue improvements.

There are even opportunities to drive news cycles with unlocked insights. Companies such as LinkedIn can use information gleaned from their web server farms to build press releases such as:  “Top Ten Phrases Recruiters Want to See” or “Top Ten Job Growth Areas in the United States”.  These kinds of press releases are interesting to local newspapers, bloggers and media outlets, especially if there’s a unique angle relevant to readers/viewers.

There are plenty of companies turning to outside firms, crowds, and even their own customers for innovation. And there’s certainly nothing wrong with any of these approaches. However, those approaches may be trying too hard – especially when there’s a goldmine waiting to be unleashed just a few web servers away.

Transparency: Do Customers Want to Peer Inside the “Black Box”?

From politician salaries to calorie counts on restaurant items, “transparency” is a key buzzword in government and business circles. However, high interest in cloud computing, data warehousing “to go,” and other analogous concepts beg the question of whether customers really want to peer inside the black box or whether an opaque approach works best.

Increases in the call for transparency are legion. Health inspectors post food safety grades for eating establishments. Websites track lists of political campaign donors. And restaurants redesign bars, kitchens, and more to show patrons how drinks and meals are prepared. All this, in order to give customers a window into processes for product and service creation.

And to be sure, there’s definitely even more opportunity for transparency in product creation, especially in financial services. As an example, Michael Lewis’ Big Short cites how via the securitization process, hundreds of subprime mortgages were packaged up and divvied into “tranches” of investment quality. Through securitization, it was tough to estimate the contents of a particular asset-backed security. One hedge fund manager exclaimed; “I didn’t know what the (expletive) was in the things. You couldn’t do the analysis. You couldn’t say, ‘Give me all the ones with all California in them.’ No one knew what was in them.”

Creating a product with so much complexity that teams of MBAs are necessary to decipher its contents surely is a recipe for confusion. And on the ugly side of things, perhaps that was the intention. Regardless, in an age of social media where a company’s reputation can be destroyed in five minutes or less, this avenue will not ensure long term success.

However, a key question is whether customers really want to peer inside the black box. After all, investment banks had very little difficulty offloading these impenetrable structured products. Plenty of hedge, pension and even sovereign wealth funds lined up to buy these complicated products—and most with no questions asked.

In the analytics market, there’s an adage that business users really don’t care how a particular solution works, just so long as it meets their needs. And while this may be true in some instances, there’s also ample opportunity to enlighten consumers (or in this case application users) as to the “value” received through peeling back the curtain on how a product or service is designed and delivered.

What say you? Do your customers really want transparency? Do they really need to know what’s in the black box?

Reputation Management Not Needed… Until It’s Needed

Poet Robert Burns is widely credited with the phrase, “The best laid plans of mice and men often go astray.” Relating this phrase in a business context, it stands to reason no matter how much a company orchestrates activities and executes its battle plans—high-impact mistakes happen. However, in an age of over-optimization, and marketing and communications cost-cutting, “soft stuff” such as brand management, press relations, crisis communications and the like are often shelved or discarded in favor of “just-in-time” strategies. Indeed, reputation management isn’t needed … until it’s needed.

In an article from “The Observer,” John Naughton wonders in amazement at how society ever managed without the Internet. Naughton ponders a world without Google, Skype, instant messaging, and online bank accounts. And while the Internet has created boom for most of us, the rise of social media hasn’t been sweet ambrosia for all companies. In fact, with social media and Internet technologies, now company decisions and actions are mostly public, including those of front-line employees. Now, actions that happened last week, last night, or 10 minutes ago can be broadcast across the globe in seconds, creating very dangerous challenges for company branding and reputation efforts.

In the Financial Times article “Perils of a Tarnished Brand,” authors Morgen Witzel and Ravi Mattu notice that even the most scripted and orchestrated product launches can go haywire. And even when “best-intented” marketing plans are well-executed, companies can be exposed to the ramifications of their daily operational and strategic decisions (e.g., Google in China and BP). “What affects reputations, in turn affects brands,” the authors point out.

Every employee is a brand ambassador, and brand management is no longer simply the purview of marketing managers. Even the best branding intentions can go awry when actions don’t back up corporate speak, say Witzel and Mattu.

Of larger concern however, is marketing cost-cutting trends in the name of efficiency that potentially leave brands and reputations exposed.

Robert Mabro, Honorary President of Oxford’s Institute for Energy, describes this problem in a letter to the Financial Times. He writes, “(Companies) no longer want to employ specialists in soft matters, such as political issues and the like. When an accident occurs, they find themselves hopelessly unprepared. This of course (ends up) destroying shareholder value!” Moreover, economist John Kay sums up the problem quite succinctly, “Yesterday’s cost-savings are so often today’s corporate crisis.”

One potential solution is for companies to invest more in “softer matters” like brand, reputation, crisis and risk management. Undoubtedly, some of these considerations are tough to justify in an age of narrow return on investment marketing calculations such as cost per lead.

However, Internet and social media technologies that transmit events, news and crisis accounts—at the speed of light—aren’t going away. To succeed in such an environment, companies must invest in the softer functions mentioned above even when “payback” doesn’t appear imminent.

It’s difficult to forecast all types of crises that could occur. A much better plan is preparedness. Is your company up for the challenge?

Related: Financial Times “It Pays to Expect the Unexpected

Marketing “The Shack”

The New York Times says The Shack has been a “surprise best seller.” Published in over 30 languages around the world, The Shack now has over 7 million copies in print and ranked first on the New York Times best seller list for 70 weeks. Some might argue the success of The Shack derives from effective use of traditional and social media marketing efforts. However with most book titles struggling to sell 1,000 copies, might there be something more to the success of The Shack than just good marketing technique?

Written by William Paul Young, The Shack was intended as simply a story for Young’s children and a few family friends. After reading the book, friends encouraged Young to send his novel to publishing companies. However, after rejection by more than 15 publishing companies, Young decided his book probably wasn’t ready for a wider audience. That is, until his friends banded together to form their own publishing company for The Shack.

In an attempt to not spoil the plot, The New York Times writes that The Shack is a “slim paperback novel … about a grieving father who meets God in the form of a jolly African-American woman.” And while the content of the book have made for lively debate on talk shows, religious forums and book clubs, Mr. Young states that The Shack was simply meant as a metaphor for “the house you build out of your own pain.”

With a mere $15,000 for an initial publishing run and a “starter” $300 marketing budget, The Shack became an instant phenomenon as buyers passed the book to friends, and others purchased cases through Young’s website.

And while controversy may have helped create awareness of The Shack, one would be remiss not to appreciate marketing efforts for the best seller.

In addition to use of traditional media to spur sales such as book signings, speaking engagements and website development, publisher Windblown Media also made extensive use of social media.

For example, a specific and bold call to action asked readers to visit The Shack website and share their own personal experiences after reading the book. In addition to an author blog, website visitors could join a Flickr group to share pictures of “beauty,” sign up for email updates, become a fan on Facebook and follow author tweets.

Also of significance, the marketing campaign accessed terrific word of mouth techniques, as the author encouraged readers to share the book with friends, blog about The Shack, buy books for women’s shelters and prisons, and even write reviews for local papers, magazines and websites.

The results? While some authors clamor for even 10 reviews of their book on Amazon.com, The Shack racked up over 4,000! A cursory review of Amazon’s listing shows that the book definitely struck nerves, sparked controversy, and deeply touched readers.

While Paul Young and Windblown Media certainly employed traditional and effective marketing techniques to maximum effect, selling seven million books is no small potatoes. Indeed, The Shack reached a tipping point then quickly accelerated to become an international sensation.

The question remains, why? Is success of The Shack due to marketing, or is there something more?

I’d love to hear your thoughts!

The Simple Minded Effects of Social Media

dunceNeuroscientists have shown in study after study, that multi-tasking isn’t helping us be more productive, but in fact, is making us dumber. Are social media tools, with their promise of instant connectivity, notification, and collaboration adding fuel to the fire?

There is a lot of excitement about staying up-to-date and making personal connections with new media tools and applications (think: Facebook, Twitter etc). However, in our attempt to multi-task and keep up to speed on everything we deem relevant, there’s a potential dark side—the dumbing down of our brains.

The Atlantic, features an article titled, “The Autumn of the Multi-Taskers”. In the article, author Walter Kirn, discusses the stress we place on our minds and bodies when we attempt too much multi-tasking with social media tools, Blackberry’s, IM and more.

For example, Kirn notes that through the use of functional magnetic resonance imaging, scientists have discovered:

“Multi-tasking messes with our brains in several ways. At the most basic level, the mental balancing acts that it requires—the constant switching and pivoting—energize regions of the brain that specialize in visual processing and physical coordination and simultaneously appear to shortchange some of the higher areas related to memory and learning.”

We like to pride ourselves on the ability to keep up with it all. We ask ourselves, “Why can’t I be on Outlook, have my IM application open, pick up the phone, read a business magazine, and have Linked In and Facebook running all at the same time?”

We’re taught that multi-tasking is the wave of the future. Do more with less. Keep up on everyone and everything. We tell ourselves we can do it all.

Neuroscientists, however, would disagree.

Kirn’s article mentions a study where two groups were asked to sort index cards. One trial group sorted in silence, the other had the same task but also was required to listen for specific tones from a grouping of sounds. At the end of the experiment, both groups sorted the cards properly, but the multi-tasking group couldn’t remember what “exactly” they were sorting.

As social media technologies (i.e. RSS, social networking and web applications, micro-blogging etc), become more prevalent and adoption rates climb, it seems we’re staying more connected with our communities and world, but forgetting half the stuff (perhaps purposefully) pushed to us via these technologies.

Our brains are out-tasked and overloaded—and yet we often look for more opportunities to cram additional information into our heads.

Sometimes, this pursuit of an “always-on” world translates into ill effects for our bodies. The article continues;

“Certain studies find that multi-tasking boosts the level of stress related hormones such as cortisol and adrenaline and wears down our systems through biochemical friction—prematurely aging us. In the short term, the confusion, fatigue and chaos merely hamper our ability to focus and analyze, but in the long term they cause (our brain) to atrophy.”

Despite the provocative title of this post, it is not my intention to indict social media technologies, or the use of any other technology such as cellular phone, PDA, instant messaging and the like. The real issue of concern is lost focus and effectiveness when we use too many of these technologies at the same time.

I believe the best course of action is a careful balance of the use of these value adding technologies with our innate ability to capture, process and store information.

We’re not a machine, but I wonder sometimes if we think we are.

Behavioral Targeting – Where’s the Fine Line?

privacyBehavioral targeting has caught the attention of the US congressional leaders, as privacy advocates grow concerned with the tremendous amount of web data collected by internet businesses such as ISPs and search engines. Consumers, lawyers, congressional leaders, and businesses are now opining regarding necessary disclosures and the appropriateness of targeting offers/advertising based on web visits and/or queries.

When it comes to behavioral targeting (using clickstream data), where is the fine line of benefit vs. “big brother”?  read original column