Strategy and the Cloud — Delegate, Delegate, Delegate
What if I told you that you could work less, delegate more, and increase customer satisfaction with data exponentially? What if I told you that you could probably accelerate the time it took to deliver data products to the customer in doing this? What if I told you that you could do all of this while increasing the technical skill sets across the company, which would allow the IT team to focus on more valuable activities?
This has all the makings of a late-night infomercial where the narrator hops into his Ferrari and speeds off. Cue the voiceover offering a course that you can buy for $10,000 to learn how to do all of this and more.
The reality is that this is possible if you deploy the right strategy with your stakeholders. Not only is it possible if you do it right, but it should also be easy (conceptually) and embraced across the organization. Here is why.
Data Can Always Do More
Regardless of the state of an organization’s data, there is almost always a demand for data to answer that next question. The paradox with data is that the more questions you answer, the more questions will be generated. The vice president that wants a Profit and Loss statement for her business unit will be satisfied with that P&L statement for roughly 0.01 seconds. She will then want to know ‘why.’ Why are the costs what they are, why are revenues down, what role do cyclical vs. secular trends play in next quarter’s budget, how can we save 10% with better inventory management, and the list goes on.
The problem with this is three-fold. One, no IT department can position data, build data models, and serve up BI reports answering all of these questions in any semblance of a timely manner. It takes a couple of seconds to ask an intelligent question, but it can take months to serve up the answer. Two, this problem is often multiplied across several departments and teams. Operations typically have multiple business units asking questions unique to their business unit. You then have a supply chain, accounting, corporate planning, human resources, health and safety, public relations, and the C-suite, all with their unique set of questions and end-of-year targets that need to be met with data.
The third issue is my favorite. Traditional IT gets overwhelmed with data requests, so they take the only logical route and apply value to each request. In theory, this is how every business project should be assessed. In reality, this becomes a muddled mess. I know because I spent half of my career in operations pencil-whipping numbers to get IT’s approval. Instead of knocking down the data projects with the most economic impact, the projects go to the team with the best salesmanship. The other flaw in this approach is that using data to solve problems starts with a hypothesis. I think something is an issue, so I review the data. If my hypothesis is invalid, the team has to start over with new data.
The solution to this is simple. Make the organization do the work.
Processes Over Answers
The suite of tools that Hashmap works with every day to help clients migrate data into the cloud allows companies to rapidly position data in the cloud. Whether it is Fivetran, Matillion, ADF, Talend, HVR, Qlik Attunity Replicate, or another ETL/ELT tool, the ability to position incremental data for the organization consumes is now done with a few keystrokes in an afternoon. We have done cloud enablement projects where one of our team members was literally able to migrate an entire business unit’s data off of an Oracle server and into the cloud in less than three days. Fifty percent of that time was spent getting him access to the systems.
This is where IT can make massive strategic gains in how the organization interacts with data. Traditionally, IT would receive a request to make data available for a business unit. It would go on the project list based on its value. It would be evaluated at a weekly or monthly committee meeting of IT and business leaders. If and when the request was addressed by IT, it would be broken out like a project with weeks of assessing and testing as IT worked to understand the business need around the data, extract the data, transform the data, and then load the transformed data into a data warehouse for the business to use. The time between request and resolution was often such that the business would create some shadow-IT process in the interim to circumvent the obstacle.
This is where being strategic with the tools that enable cloud data warehouses to make all the difference. First, many of these tools are designed to push massive amounts of data into the cloud. So when connecting a source system to the cloud, aim to load any and all data. Second, cloud storage is ridiculously cheap. At $25 per terabyte, you can error on the side of loading in more data than necessary. Having almost all data (assuming it does not pose a regulatory or security risk) in the cloud makes answering additional data requests much easier because the data is already loaded in the warehouse.
The business challenge in this model is that while data is available, it has not been transformed in ways to make it meaningful to the business. However, this is now a challenge for the business to address IT as a business partner that can assist. It is no longer IT’s sole problem to solve. The business is also empowered to explore this data. When they are ready to transform it in meaningful ways, they can provide valuable context to IT to build out those transformations faster.
A successful cloud migration program includes a strategy that creates processes to extract and load data, position data for the business to access, and transform the data as needed. I’ve created data management processes that do not only this but also 1) apply a classification to the data, 2) obtain and document approval from the data family for its use, 3) catalog the data along with any transformations that were applied, and 4) incorporate the data within existing data models for fast consumption. We have been able to design this process so that it averaged 2–3 weeks for the business to get what they needed from request to delivery.
In doing this, we delegated to the business the work of finding the necessary data, understanding the context around the data, agreeing on the source for the data, and providing initial logic around the data to be used for analysis. IT was able to step in and mature the transformations using a transformation tool like dbt.
Delegate, Delegate, Delegate More
You will find at least one power user for your company’s business intelligence tool in most teams. From Power BI to Tableau to Spotfire, most teams have at least one member that has developed at least rudimentary skills in writing SQL queries. IT departments should work to identify these individuals and invest in their skills. By enabling the business to build their own tables, views, and data models, IT can focus on maturing data models, building out analytical tools that are more predictive, and spend more time leveling up the organization’s technical skill sets. By creating citizen data engineers and citizen data scientists within the business, IT professionals can continue migrating up the IT value ladder and work on projects that can give tactical and strategic advantages to the broader company.
Final Thoughts
As you build out your cloud enablement strategy, challenge your team to identify areas where the business could be doing the bulk of the work, where processes could replace projects, opportunities to offer resources to educate the broader organization, and where IT can refocus data engineers for higher-value analytics.
If you would like assistance in building and executing a cloud strategy, please reach out to us. We can provide consultation services or work side-by-side with your teams to accelerate your cloud journey.
Hashmap’s Data & Cloud Migration and Modernization Workshop is an interactive, two-hour experience for you and your team to help understand how to accelerate desired outcomes, reduce risk, and enable modern data readiness. We’ll talk through options and make sure that everyone has a good understanding of what should be prioritized, typical project phases, and how to mitigate risk. Sign up today for our complimentary workshop.
Other Tools and Content You Might Like
Feel free to share on other channels, and be sure and keep up with all new content from Hashmap here. To listen in on a casual conversation about all things data engineering and the cloud, check out Hashmap’s podcast Hashmap on Tap as well on Spotify, Apple, Google, and other popular streaming apps.
Scottie Bryan is a Delivery Manager with Hashmap, an NTT Data Company, providing Data, Cloud, IoT, and AI/ML solutions and consulting expertise across industries with a group of innovative technologists and domain experts accelerating high-value business outcomes for our customers. Connect with Scottie on LinkedIn.
