A lot of companies are now focusing on data-driven architecture but often leave this in hands of sponsored teams who are always in focus to deliver value in a fixed time to their product or business owners.
The question is whether this is the right strategy for achieving the goal?
From my perspective, the answer is “Yes”. However often teams channelized their energy in delivering poorly defined MVPs in timely basis without focusing on building the right foundations. This is a beginning of a vicious circle, companies need to avoid this.
Data Architecture can look like a sandwich below, filled with juicy patty, pickles, cheese and sauces in between(speaking figuratively)
Usually Data generating & consuming systems can be same or different, but they tend to drive overall business of the company in terms of interaction with customer, products, ability to serve or partner with other businesses.
I would mention few basic steps which can be taken by any organization as direction to move towards Data Centric Enterprise, and build the right foundations for Data Driven architecture:-
- Build your team ‘Right’:- You must have right team which comprises required knowledge , expertise and passion towards goal. This includes Business Acumen, Domain experts, Data Technology experts and most important ‘Smart’ Delivery or Technology experts who can manage Business and Delivery Teams effectively. This team is your ‘Chef’ who prepares this sandwich. Don’t assume that in-house chefs always knows the new recipe, at same time external chefs might not know your kitchen and customers. Find the balance!
- Build framework for all type of data pipelines: Going by legacy technologies, most companies have established basic data pipelines to move data from point ‘A’ to point ‘B’ in their data platforms. But use cases are changing everyday in industry. Establish all type of data pipelines , ingestion framework and foundation patterns whether it is a real time streaming, establishing self service ingestion mechanism , social media analytics or handling any such complex data patterns. Be Ready! Don't expect Business to invest in technology.
Enable all concepts and patterns like messaging , real time streaming, alerting and notification platforms(if you don’t foresee such needs now then I must say business strategies need to be revisited , however I am sure it is a matter of time where business will come up with use cases soon), naming few existing technologies in these areas :- Kafka Streaming, Spark Streaming, Flume, Graphite, Splunk etc.
Find smart ways to manage small scale tech initiatives resulting to large Business scale projects. This is similar to restaurants trying out new recipes to customers for free take out or serving new complimentary dishes before rolling out new cuisine item on the menu.
Investment into new technologies has become least expensive now in the industry then ever. This is due to market moving towards open sources, free developer licensing model in new tools and platforms , cloud transformation has helped further.
3. Embrace thinning Lines between OLAP & OLTP :- Due to extremely vast variety of data sources and required agility, companies can no longer rely only on OLAP (traditional DWs) for all analytics as modeling all type of sources and topics in DW is expensive both in terms of time and money. Secondly lot of use cases are temporary in nature, which might not require you to build full scale OLAP domain. So look for hybrid solutions to federate data between existing OLAP and OLTP systems in your ecosystem. Data virtualization / Data federation techniques can be explored and applied here with very low cost and time to market. Start building DaaS model ‘Data as a Service’ for your applications, this is one of optimal way to prepare your master data and expose it as Rest APIs to integrate easily with Microservices architecture.
4. Move to Cloud :- Start your journey towards cloud if not yet started, I understand this requires some investment & strategic changes but it will help in long term. Going towards Serverless and Containerized solutions will help companies in establishing agile and low cost solution model without investing too much on infrastructure. Cloud brings lot of disruptive data technologies in terms of storage, computing, processing at your service in low cost which you can quickly adapt and deploy. All three major cloud players, Amazon, Microsoft and Google have amazing stack of services included in these areas.
This path also reduces overall operation and administrative costs which are incurred by hosting on premise big data and other data platforms.
5.Partnership with Business to create ‘Data Driven’ Culture:- This is one of the most important step which is often ignored. When you work with Business teams they are not aware about power, usage and value of technologies & architecture whereas IT teams are not aware about the real Business goals. Build a partnership model between IT and Business teams to ensure ‘Data’ is served & consumed in right way . Don’t ask business what they want, ask them about their Business and Financial goals .Prepare and align joint ROI Model which can help create measures of defining common success for all teams for Data driven projects.
As data analytics is becoming widely used into the way organizations work today , it’s evident that data driven architecture is necessary to create and grow the data-centric enterprise. Technology leaders need to embrace this new challenge and revisit their roles, functioning methodology and be agile in adapting to this competitive environment.