There isn’t too much dispute that the ‘Big Data’ is the most used/misused IT jargon of this decade. What is Big Data? Everyone evidently has their own understanding of Big Data. All are not wrong, nor none are incorrect. Clear enough? Apparent to me, there is no real discernable common understanding.
In my risky attempt of profound, I present a quote from Charles Bukowski: “The problem with the world is that the intelligent people are full of doubts, while the stupid ones are full of confidence.” I find this quote personably alluring because I am easily confused and this sentence gives me an out, sort of.
Despite my insecurities, wouldn’t common sense suggest that a person with higher intelligence has a higher capacity to retain and process data? And, consequently, shouldn’t Intelligent people be more confident about their opinions? If you believe in Charles Bukowski’s assertions then the so-called intelligent people with doubts must live in a paradoxical realm of common sense, of sorts.
During my formative years, I learned about the concept of paradox as a result of watching too many sci-fi TV shows, and few fictions that didn’t test my short attention span (thank you, Kurt Vonnegut). My first academic encounter with Paradox was very disappointing. It didn’t have anything to do with time travel or multi-dimensional quantum leaps, rather it was something called a Gabriel’s Horn paradox. Whether you believe it is an actual paradox or not, I just thought it was stupid. Just try to explain this to any 10-year-old and see what he thinks of it.
In the ensuing years, to preserve my sanity, I succumbed to my own notion of paradox pylons. This to me was the basis for all my accumulated realities, and how I learned to ignore them as a way of keeping the zeitgeist of my sanity. Should I continue to use ignorance as a way to deal with the confusion of having too much information? By nature-or-nurture, I have a set of experiences that informs me about how to interpret the realities as I encounter them, with strong preferential bias.
Back to the actual topic, many folks write the conclusions first, and as an afterthought, use data as a way to substantiate their beliefs. The phrase “lies, damned lies, and statistics” was made popular by Mark Twain and it’s an apt one for traditional BI. We tend to cherry-pick the data that tends to reinforce our own bias. In a way, these BI tools create personalized echo chambers.
Most folks who have been in the business of working with data are fully aware of these bias tendencies. When these folks become the decision-makers themselves, they become skeptics of the hypothesis based on ‘trending’ data. They know that there are multiple streams of ‘trending data’ where one can pivot and mix different dimensions to support whatever bias agenda of interest. This becomes an itch that can’t be scratched. Or, in my own personal experience, a cause for insomnia since I didn’t have an option to ignore the nagging doubts.
The brave new world is dawning as the AI becomes a new catchphrase. However, few understand what this really means — that includes me, to be fair. People get that AI with ML/DL needs lots of data to live-in a dynamic decision-making paradigm rather than traditional BI-based projections — which is kind of like driving forward by looking at a rearview mirror. What occurred in the past is no guarantee that it will happen in the future. If the last 5 miles were a straight line, then do you trust that the next 5 will be the same?
The DL/ML may not provide a long view of the future, but an ability to see the near future is critical in most businesses. As a leader, we strive to understand what we are doing, how we are doing, and what challenges are ahead. The traditional BI has been promising these capabilities since the beginning. And it did work for some time when the data source is consistent and limited.
In 2021, everything is the data source, down to a pill that has an embedded chip to notify the caretaker when the patient takes a medication — it was a bit ahead of its time, but you get the point. These data will drive the business forward, but the business needs to look ahead in near real-time and act accordingly. Further advancement in the acquisition and processing the diverse and dynamic data with AI will usher in some new interesting use cases.
