Gartner defines dark data as “information assets organizations collect, process and store during regular business activities but generally fail to use for other purposes.” Similar in concept to “dark matter,” where invisible forces affect visible celestial objects, data collected by companies goes unnoticed, unused and generally forgotten. However, businesses are missing significant business value by letting dark data go by without pause and careful attention. In fact, by capturing and using dark data to their benefit, companies can better meet customer needs and, as a result, get a leg up on the competition.
Maria Konnikova writes for the New Yorker and is also author of Mastermind: How to Think Like Sherlock Holmes. In a Big Think article, Ms. Konnikova tells one of her favorite tales. Holmes and Watson are bantering back and forth about the difference between seeing and observing. In the vignette, Holmes says that Watson sees the world around him, but does not take time to observe things such as how many stair-steps he took to climb the loft. “There are seventeen steps!” Holmes exclaims, chastising his counterpart for charging up the steps without actually taking time to document his surroundings.
In the same way, most businesses have a pretty good handle on their operations from their reporting systems. These companies know the customers, products, suppliers, and partners that make up their industry ecosystem. However, like Watson, plenty of companies are not seeing the complete picture—in essence they have seen, but not observed.
That’s because businesses with a focus on operational reporting and analytics are in fact seeing the daily ebbs and flows of their business, but missing a whole new level of insight available from data they probably already possess but do not use. Getting back to the original Gartner definition, data is often called “dark data” because it is deemed to have little value to the enterprise.
What are some examples of dark data? Objects such as hard-to-read text files, video, audio, web logs and more comprise the world of dark data—really anything in a raw format generated from devices, sensors, and computers. Sometimes this data is in JSON format; other times it will be clunky audio or video files with missing metadata tags. Mostly it is in proprietary binary file formats such as those generated by manufacturing equipment or sensors.
When captured, stored, and analyzed, these dark data sources can drive additional value for your company. Here’s an example: one high tech manufacturer has found significant benefit to capturing sensor data from its manufacturing line in real time. In days prior to the instantiation of sensors and a big data system to capture and analyze (dark) sensor data, this manufacturer would only recalibrate its manufacturing systems during various downtime windows when the whole line was shut down for maintenance.
By capturing and analyzing sensor data as it flows by, the manufacturer found that a single machine—out of hundreds—was out of spec due to a faulty cable. The manufacturer would never have found this anomaly prior to the maintenance window without the analysis of real time sensor data. By speeding up the ability to detect deviations within seconds instead of days, and calibrate manufacturing equipment faster than ever before, product quality has increased and the likelihood of shipping bad product has drastically decreased.
Whether we realize it or not, there are countless sources of so-called ‘dark data’ in our business just waiting to be analyzed. These data sources are mostly messy and raw, and they take time to wrangle, but there’s tons of potential benefit to analyzing them.
Getting back to Sherlock Holmes, author Maria Konnikova asks a compelling question that Holmes would ask all of us: “How much do we miss (by not observing), that would actually make a difference?” We must go beyond letting dark data escape to either tape or cold storage in the cloud. It’s time to go beyond seeing the things we’ve passed by hundreds of times, to actual observation. By capturing and analyzing our own dark data, we’ll be in a better position to solve the puzzle of who our customers really are, what they’re buying, and what they actually think of us.
Originally posted in ForbesBrandVoice: http://www.forbes.com/sites/teradata/2016/01/06/dark-data-take-a-page-from-sherlock-holmes/