Re-Imagining Data Architecture — Part 1
Data Architecture Design Should Be Technology Agnostic and Abstract For Future Innovations
In a cloud-ready market, readily developed and deployed data models are showing limitations in computation and data munging.
The chaos of data and its growth as silo’s within every enterprise is known to us. This architecture needs to be re-imagined in the new world. The current model of data munging is through business rules embedded within ETL programs built in Informatica, Datastage, AbInitio or SSIS. These systems are all built to work on fixed length data structures that are well defined, which may not be the case in the next iteration of data evolution.
In the optimal solution model, if we are to bring new formats and forms of data, and have scalable and flexible data architecture processing models, we need to think new and different.
Can we rectify the data structure prior to making another migration of any sort in the Cloud or On-premise? Can we solve this riddle? Is the architect finally going to have their day?
In my opinion, having survived this massacre for close to three decades, the answer is: yes. We can have 500 columns or 5 columns, but we need to capture it all and bring it across. Here is where the glitch in the data architecture occurs. We have always extracted the data as attributes needed by business users, and have ignored attributes based on requirements.
In the new world of cloud, containers, data pipelines, neural networks and machine learning algorithms which are all the part of our toolset, we will need to evolve data architecture processing too. In order for that evolution to occur, we have to identify the risks, pitfalls and recovery options.
This is the first in a series of posts on data architecture re-imagined for the cloud and containers. The new model of the prcoess will force us to move away from being SQL centric and add more integration of microservices and pipelines, which will define more complexity and at the same time increase hortizontal scalability.