In Big Data Endeavors, Don’t Neglect Softer Business Skills

With technical skills such as Java, C++, Python and more in high demand for “Big Data” analytics, it seems like softer business skills such as speaking, writing, planning, leadership, negotiation etc. are falling by the wayside. But the ability to communicate, relate and navigate throughout an organization—so called “softer skills”—are especially needed to propagate analysis and communicate the impact of data-driven decision-making.

Courtesy of Flickr. By coryccreamer

In 2012, cloud computing blogger David Linthicum penned a short piece explaining “3 Winners and 3 Losers in the Move to Big Data”.  In the post Linthicum identified one “loser” as data warehouse and BI specialists, presumably because these folks were accustomed to using languages like old-school SQL and supporting “legacy BI” systems.

It’s interesting that as we find ourselves nearing mid-2013, those “legacy” skills of writing for and supporting various BI tools and relational databases are not going away. In fact, the opposite seems true as open source programmers seek more ways to make projects SQL-like to access various distributed file systems, NoSQL and NewSQL data stores. And while the development of SQL-like interfaces helps the business analyst utilize some of these newer platforms, business skills seem to get short-shrift in the equation of making an analytics program a success.

It appears the burgeoning role of “data scientist” intends to bridge the gap between technical skills and business acumen.  An IBM blog states that while the formal training of a data scientist should include an understanding of computer science, applications and ability to write in various languages, they also need to have business smarts.  Thus the data scientist role must marry technical skills with “the ability to communicate findings to both business and IT leaders in a way that can influence how an organization approaches a business challenge.”

It would seem that bridging the technical and business acumen gap with the data scientist role is an excellent idea. However as many articles on this site point out, data scientists are in high demand and can cost an organization a pretty penny. And at this point, there just aren’t that many data scientists available on job boards, or willing to move out of Silicon Valley.  So it appears that while there are plenty of employees with technical skills, and line of business leaders that understand the inner workings of the enterprise, there’s still a gap that needs bridging. What’s a company to do?

While it’s debatable whether a business analyst can be taught the necessary technical skills to become a data scientist, we can definitely ensure that we don’t neglect softer business skills in the evolution towards a data-driven organization. For example, there are universities that offer classes and executive course work on negotiation, communication and selling skills. In addition, there are programs available such as Toastmasters that can teach leadership and public speaking skills.

Need help writing? Your local university likely has coursework and workshops to improve business writing for proposals, sales briefs, whitepapers and more. Finally, there are too few employees that can perform “critical thinking”, or the ability to conceptualize, analyze and then evaluate various streams of information. Coursework from universities across the globe can also assist in this area.

What say you? Are better business skills needed for analytics professionals? If so, what are those skills? Finally, how would you recommend developing an action plan to “perform a business skills upgrade”?

How Mobile Operators are Mining Big Data

Mobile phone operators have long mined details on voice and data transactions to measure service quality, place cellular towers in optimal locations and even respond to tariff and rate disputes among various carriers.  But, that’s just scratching the surface for getting value from mobile data.

Image courtesy of Flickr. Milica Sekulic.

Call detail records (CDR) for mobile transactions are particularly interesting for analysis purposes.  According to a Wikipedia entry, CDRs are chock full of useful data for carriers including phone numbers for originator and call receiver, start time, duration, route, call type (voice, SMS, data) among other nuggets. It’s not unusual for mobile operators to mine 100 terabytes (TB) and up databases to optimize networks, strategically position service personnel, perform customer service requests and more.

And carriers are also starting to discover value in performing social network analysis (SNA) in relational databases and MapReduce/Hadoop platforms to analyze social/relationship connections, find influencers, and –if directed by government authorities—even perform crime syndication tracking or terrorist network monitoring.

While the types of analysis listed above are becoming commonplace, mobile phone operators are learning a lot more from “Big Data” analysis of everything they’re capturing.

Financial Times writer Gillian Tett explores some of these innovative approaches in a recent article (registration required). Tett notes that with mobile phone subscribers topping out at 2.5 billion subscribers in emerging markets alone, that mobile carriers, behavioral scientists and governments are learning more about “people’s movements, habits, and ideas.”

For example, Tett cites the 2010 Haitian earthquake where aid workers alongside researchers were able to “track Sim cards inside Haitians’ mobile phones.”  This in turn helped relief agencies analyze where populations dispersed and helped route food and medicine to where it was needed most.

Analyst firm IDC notes that smartphone sales are flying out the door at the tune of 400 million a quarter. With the rise of smartphones, there are also more mapping and location based applications online too. In fact, when billing, use, location, social networks, much less content accessed and more come into view, there will be little left to the imagination to complete a picture of who you are, where you’ve been, what you’re doing, and where you’re predicted to go next.

These types of rich information will be accessed for customer, corporate and societal benefit. However, there’s also ripe potential for mis-use. The key questions are – is this much ado about nothing, or a data collection spree with an unhappy ending?

How Much Big Data is Too Much?

With storage costs plummeting and sophisticated software approaches to mining Big Data, it appears that it is increasingly cost effective for corporations and governments to keep all types of data, even those previously discarded.  However, how much “Big Data” should corporations, entities and governments keep online or archived, especially when “Right to Be Forgotten” debates are swirling?

Image Courtesy of Flickr

Like it or not, all kinds of data are captured every day. James Gleick in “The Information” sums it up nicely;

“The information produced and consumed by humankind used to vanish—that was the norm, the default. The sights, the sounds, the spoken word just melted away. Now the expectations have inverted. Everything may be recorded and preserved at least potentially; every musical performance, every crime, elevator, city street, every volcano or tsunami on the remotest shore…”

With petabytes of storage and virtual machines available in the cloud on a pay per use basis, and on premise storage costs dropping like a rock, it’s conceivable for companies and governments keep every image, video, recording, keystroke, and web generated data type. And of course, all these data are of little use without techniques to mine and perform information discovery. Fortunately BI and data warehousing technologies have worked wonders over the past thirty to forty years for data that needs to be organized, and we have MapReduce/Hadoop to assist in assembling/analyzing an organized data garbage dump.

There are two consequences of this data deluge.

For individuals, there is the feeling of drowning in a sea of overwhelming data of which it’s difficult to manage much less scrutinize. Novelist David Foster Wallace called this scenario “Total Noise” to coin the feeling of drowning in a deep pool of too many tweets, posts, phone calls, podcasts and more. And because this total noise causes “information anxiety” for some, there are plenty of people deleting social media accounts.

And there is a second consequence of this data deluge. Since everything that can be captured is in the process of being captured, there are certainly privacy and security concerns. Our likes, rants, passions and partialities are recorded online and archived offline in perpetuity. These concerns have fomented potential privacy legislation such as the EU’s “Right to Be Forgotten” where digital providers—upon request—will need to cull digital references owned by individuals.

These consequences then beg the question, how much Big Data is too much? What should be kept for corporate reasons (to serve customers better, sell more products, optimize business processes etc)? What should be kept for governmental concerns (tracking bank flows for money laundering, watching for potential terrorist activity, monitoring fringe groups that don’t see eye to eye with government officials)?  And with pending legislation such as “Right to be Forgotten” considered in statehouses across the world, is it more hassle than it’s worth to keep all this Big Data, especially if there are financial penalties for not complying with legislation?