Will Data Be The Next .com Collapse?
Perhaps the title is a bit dramatic. The .com bubble was a different beast entirely but there’s some similarities happened in the data space that we can discuss.
I have been involved in data for over a decade and have seen the industry scramble to find it’s feet in a high demand environment. Hype and buzz drove a frenzy of noise and demand for something that, let’s be honest, in the early days, the average person out there didn’t understand the depths of, nor did they know what they wanted, let alone know what they needed! BUT they needed it because the hype around it said they did!
Data promised the world! It will solve all of your business problems! You will become an 8 or 10 figure business overnight! But only if you can navigate the mystical road to data enlightenment and harness the invisible power of the data matrix, much like Neo in the Matrix movies.
In order to do this you must invest 8+ figures into tech stacks and data teams without question! But wait! Data per se wasn’t a mainstream industry in its own right yet, so we must first set up data capability and train employees into data engineers, architects and analysts who can operate in this new world.
Later would come the “data scientist”, a new title for what we once called statisticians, made into a buzzword around 2012 and the position uplifted to the upper echelon of the data lords. They who achieve this ranking are equal to the likes of Gandalf the Grey! With a single shake of their staff they can command the power of data into magical intelligence never seen before, intelligence that will finally provide the answer to the unanswerable... Perhaps even the meaning of life itself?!
We saw buzz phrases like “big data”, “ data-driven decision making”, “data mining”, “business intelligence,” “predictive modelling,” , “data warehousing” and ‘data lakes” pop up over night.
Industry rode the bumpy wave of hype, never game enough to slow down, let alone stop and consider anything for too long. Mainly for fear of being left behind on the scrap heap, to fight it out in the obsoleteness of data inferiority. Much like the scrap yard scene in the 2004 movie iRobot.
Big tech giants rose to dominance and drove much of this fast paced hype. With large scale investment in server farms, where immense amounts of data could be captured, stored and somehow used. Not without proving its usefulness and benefits! We’ve seen the power of this data used everywhere around us.
Data platforms and systems were implemented in haste and often were a square peg in a round hole in the broader IT ecosystem, creating additional technical debt. But let’s not allow that to slow our ever growing lust for more data! Five nines? Who needs it?! Out the window to make room for more data, we’ll deal with issues if and when they occur!
Agile Development… The data journey pushed this to a whole new level! Some may call it “make it up on the fly without any kind of plan, just get us data and now!”.
Technologies quickly emerged to help us solve many of the problems with becoming data driven entities. Making it quicker and easier to extract data and turn it into fancy looking visualisations and insights. This resulted in fixing 1 issue, only to create many others. Mainly, it drove more questions from business and a thirst for more “meaningful insights”, which meant more data and resourcing to manage it. An endless cycle of supply & demand.
Then Covid struck. This sent the data world into a spiral of endless demand for more data and better insights. Demand for data expertise grew exponentially as did the renumeration demands from anyone who has any kind of data expertise. This led to a bidding war to secure good data resources.
Where does it end?
Whilst i’m ever the advocate for good data, i’m also exactly that… an advocate for GOOD data. Not terabytes of completely useless data, not dashboards that look amazing but tell me nothing I don’t already know and certainly not a “model” that claims to solve all of the worlds problems, only to need constant “adjusting” to fit the desired narrative.
In the rush to become data literate and capable, we’ve perhaps lost sight of initial requirement, which was to use data to assist industry move forward. Instead, we’ve begun to step into a dystopian state where data is given a level of grandiosity and ultimate power to be running industry. Where all decisions revolve around those who can harness its mystical powers and spin the narrative.
At some point industry will come to a realisation that data isn’t the be all and end all they were expecting and begin to rationalise their investment. The key will be to manage a soft rationalisation, rather than a complete collapse and withdrawal of support. After all data has proven it’s worth 100 times over.
Where to from here?
Cue the year 2023. Much like the IT industry in the early 00’s, a rationalisation of the data industry is inevitable. Endless spending can’t go on forever. Inflated hype around data and it’s magical powers has to deflate into a managable level of reality. Data debt has become a real concern, that has to be managed and incorporated into our data strategies, or we face the real prospect of a data debt level that is beyond salvageable.
Let’s be frank, the rationalisation is already quietly occurring. Big tech have been shedding staff in a bid to adjust to the current volatile economic environment. This will impact the data realm. The term “big data” is somewhat past it’s prime, some even say it’s obsolete. I prefer to think the data journey has matured and “big data” has evolved into the conceptual realm in our broader data strategy, rather than the tangible data assets as it once was.
With the rise in prominence of cloud platforms, has come a 3rd party cost to hold and process every byte of data. While blob storage is dirt cheap, moving and processing of data comes at a unit cost. As we would expect and rightly so, Providers are setting their pricing models to maximise their revenue while maintaining sustainable growth. This is forcing us to rethink our data strategies. No longer can the race be focussed on accumulating and crunching as much data as humanly possible, in order to reach the upper echelons of big data status.
Rather, focus is shifting to well thought out structured data holdings. Enough to serve the need of the business, allowing for key use cases, such as targeted analytics, while being conscious of processing and storage costs, the cost and scale of in-house data capability and the threat of unmanageable data debt.
Cloud providers and other technology companies are shifting focus onto addressing the broader data ecosystem designs and the accessibility and governance management issues, that have plagued us through our data journey over the last decade or more. They are working with us to design manageable and scalable ecosystems that serve our specific business need, without being over the top and unnecessary.
The future
The real gains will be internally. Industry needs to bring data back to reality and get back to the basics. Identify the need, assess the cost vs benefit and make calculated decisions on investment in data capability.
Moreover, there needs to be a realisation that despite the advancements in modern data ecosystems, they still do not offer data teams greater visibilty of their data in their own right. Hard work by data experts is still required to harness the data in the systems.
Nor do they facilitate better communication and transparency between data teams and stakeholders. Old fashioned stakeholder management and liaison is still required to identify the need and translate that into a valuable data asset.
Data teams need to “pick up the phone” and talk with their business counterparts. There has perhaps been a misnomer that data teams had to become experts in every aspect of the business in order to produce their outputs. This in reality has the potential to lead to biased data and insights. Experts run the business, data teams should leverage off this expertise to transpose data into meaningful insights. Keep the separation of duties in place, it serves a valid purpose.
“Quick wins” Not a term i’m fond of! Too much focus on these stifle long term strategies that ultimately drive success, but it has its place. Organisations should have a good mix of quick wins and long term strategies in their overarching data strategy.
And finally. A formalised data strategy. If you don’t have one you need one! It should be developed and agreed upon between data teams and the rest of the business. Be realistic and don’t shoot for the stars. Identify what the business wants and then refine that down to what the business needs. There’s often a significant difference between the two!
