“Big data has turned the ‘Single Pane of Glass’ panacea of yesteryear into the ‘Single Glass of Pain’ problem of today.”
Your machines have a lot to say, just like your friends on social media. Are you listening? Once you learn to, you’ll notice some similarities between machine messages and the tweets of your most enthusiastic social media friends. Just like people, machines will tell you when they’re feeling normal and when they’re having problems. Many times they’ll reveal their most intimate secrets to you, in 140 characters or less. But are you listening?
The first requirement of gaining productive use of your machine data is to actually have the data. Herein lies the first phase of machine data mastery...
Phase 1: The Data Gatherer
For many years machines left us perplexed by a lack of data, particularly when they misbehaved or failed. This drove industry leaders such as David Henke (SVP of Yahoo Operations back in the day) to remind us of the most significant basic lesson in managing information technology: “what doesn’t get measured, doesn’t get fixed.”
The “Data Gathering” movement and the premise of “the more data, the better” forms the foundational aspects of today’s big data movement. With machines, sensors and controls getting better and better at producing data, our ability to effectively gather it becomes the primary goal of this early stage of the game.
Once we become proficient gatherers, the results of doing so quickly drives the key question that defines the next phase of our mission: How the heck do we make sense of all this raw data ?
Phase 2: The Data Interface
Eureka! Just like an old oil well hitting its first gusher, early flows of data tend to quickly spew about uncontrollably. Logs and dumps quickly become a wasteland of bits, and the data that was originally hard to gather now becomes the data that is hard to organize.
Herein comes the phase of dashboards, metrics and pivot tables. During this era, the “single pane of glass” becomes the new milestone to be proud of when it comes to simplifying the visualization of all this data. As a result, these new data windows enable us to better digest the information presented, in order to make better human decisions.
That said, the data keeps coming and coming. We don’t know what to do with all of it. We store it. We’ll worry about it later. Pretty soon stored data begins to look awfully big. So big that we come up with a creative new name for it: “Big Data.” Having only reached phase II, that’s not necessarily a compliment either....
Phase 3: Big Data Analytics
Continuously growing Big Data quickly turns your “single pane of glass” into a “single glass of pain.” Volumes shatter simplicity. Once our human threshold for digesting information is hit, big data quickly becomes big problems.
Starting with open source Apache tools such as Hadoop,and advancing towards more modern equivalents such as Spark and Flink, in phase 3 humans now submit to the human limitations in making sense of big data and now rely on powerful large compute clusters that help quickly analyze it all.
Big data analytics entails machine discovery of meaningful data patterns and trends. Done either in real time or in stored batches, analytics simplify big data by sorting through and providing a bird’s eye view into trends, that makes sense to people. Analytics is the first step in starting to make your big data work for you.
Phase 4: Deep Insights
If analytics are the gold mine of big data, than insights are the gold itself. Data insights reflect the learning lessons that are gained through the results of analytics, combined with human and machine learning. Insights are what make us and our operations smarter every day. We teach. We learn. We share. We repeat.
That said, insights without action is like revving a car without tires. The next phase is where the rubber hits the road….
Phase 5: Insightful Actions
This is where the pieces come to together. Where all the data, enagement, analytics, and machine learning turn into automated actions, based on all of the above. Through these insightful actions, our machine operations become stronger, better, most cost effective and reliable every day. They seek to catch problems before they occur, understand them as they occur, and solve them before they can occur again.
In the eyes of human perspective, the speed at which automated insightful actions execute appear to be almost instinctual in nature. A flow state.
Insightful actions are what drive true best-in-class productivity in todays machine driven enterprises. How far are you from realizing this potential? and, what’s next?
As machines “tweet” more and more, there’s also a growing difference between them and people: The average active human Twitter user tweets 1.7 times a day, while a normally active machine is more than capable of tweeting 100,000+ times a day (a real chatty machine can do that in under a second.)
Growing machine message volumes over the coming years will come to reinvent todays “big data,” with the definition of ‘big’ growing millions of times bigger than it is today. This exponential disruption underlines the importance of effectively getting ahead while you can. Doing so using the right approaches, at the right place (not just at the cloud, but through the entire vertical stack-- cloud to fog to edge device,) is of growing importance. It will prepare us for the next phase, which is…...