How Cost Management Can Tank Your Cloud Migration
Migrate and Modernize while Saving Costs
A new data warehouse is a tough sell in any organization. In the good times, an organization is focused on reinvesting in operations to grow revenue. In bad times an organization is focused on reducing costs. Justifying initiatives that don’t benefit the bottom line immediately or benefit the bottom line indirectly is always challenging.
In a previous life, I was moved out of operations/finance into the IT department to help build excitement for a cloud data warehouse. The first 30 days were like wrestling an angry porcupine. The analysts that I met with were terrified that they would lose their Access databases. Their managers and directors explained that my sales pitch for a cloud data warehouse was the same sales pitch they had heard three years earlier for Hadoop. Namely, the ‘new’ data warehouse would solve all of today’s problems and set the teams up for years to come. The finance team reminded me that this was the same sales pitch they had received for SAP BW four years earlier. That one was ironic because I had been one of the organization’s most vocal critics of SAP BW.
There were a lot of difficult conversations. However, no team’s data needs were being met which made my job easier. Everyone was unhappy, and the organization had a strong track record of never retiring legacy systems. My sales pitch quickly evolved to:
- “You aren’t happy with the last system you were sold on.”
- “Your data needs aren’t being addressed fully by any data warehouse or combination of warehouses.”
- “Let’s try the cloud. What do you have to lose? You can’t get more unhappy.”
Those three points were good enough to get nominal buy-in across most departments.
Initial Success vs. Sticker Shock
Our foray into the cloud started slow but was met with early success. We created several key star schema data models that provided executives and senior leadership with KPIs for the broader organization. We built the data models to make the underlying data readily available for the broader organization to build their own reports and dashboards. This enabled most teams to leverage the data to improve the metrics being tracked at the executive level. Several teams slowly started to embrace what we were doing. We were starting to see some real momentum across the organization when we realized we were headed towards disaster.
We had adopted a cloud strategy that assumed costs would be inherently less expensive than on-prem data warehouses, which was true from a storage perspective. However, we had not factored in the cost to compute. On-prem costs to compute generally sunk costs. They were paid for years ago when the server was bought, and the data center was built. The only costs that the organization really felt daily were the cost of time when it took a query forever to run and the time-consuming workarounds to analyze large amounts of data.
For example, I’ve heard of accounting teams that spend days running quarterly reports. Not only are the reports time-consuming to run, but often the teams have to run the query for each month of the quarter and then combine the data through an Access database to run the required reports and analytics! The organization loses time, but the risk increases due to the manual handling of the data once it leaves the source system.
The cloud had removed the physical limitations of our on-prem servers. That accounting query that took 45 minutes and timed out 50% of the time could now run in less than 10 seconds, and that query included two years of accounting data instead of just a single month. The technical limitations were obliterated, but each query run came with the cost to compute. Teams conditioned only to run what the servers could handle found that they could compute exponentially more data and get exponentially better performance. For the business, running a massive and poorly engineered query was no problem. For IT, the costs quickly added up.
Cost Management is Critical in the Cloud
We had promoted this cloud data warehouse as a way to save costs. Our architects realized we were headed towards trouble—the kind of trouble that can hijack a career or someone’s reputation.
At Hashmap, we see this scenario all the time. An organization steps into the cloud because they were promised better performance and lower costs to leadership. Then the invoices start coming in. We’ve seen organizations spending between $50,000 and $100,000 a month that came to us for guidance and help, desperate for a solution.
When migrating to the cloud, the challenge for any organization goes beyond the complexity and the discipline of migrating data to a new data warehouse and everything that entails: security, data governance, data management, change management, additional training to query data, and data quality. Setting up a cloud data warehouse isn’t a core competency for most IT organizations, nor should it be. Most companies will establish a cloud data warehouse once if done correctly.
However, if you do it wrong, you spend more money and lose more time than if you had brought in a team that possessed cloud migrations as a core competency from the start.
I recently finished a project with a client spending around $50,000 in the cloud each month. Our 45-day project realigned and reconfigured their cloud data warehouse. The internal rate of return on what the client paid us versus what was being spent for a cloud data warehouse should reach 100% in the first year!
Fortunately for me, my team’s journey into the cloud did not get as far down the road as the client that I worked with. We had not blown our budget, but we did issues along the way. We had to rework a lot of data, reengage the business, and after nine months of moving to the cloud, we had to change cloud providers. I mentioned earlier that getting the organization’s initial buy-in had been like wrestling a porcupine. The engagements telling departments that we were switching to a different cloud data warehouse provider was even worse. We had broken trust, wasted time, and wasted resources.
Key Takeaways — Hedge Your Sales Pitch, Manage Your Warehouse, Move Slow, Use a Guide, Move Fast
For an organization preparing to establish a cloud data warehouse for the first time (or a second time), here are the recommendations I would provide based on my personal experience working within an IT organization that was going to be held accountable for that cloud data warehouse, and my time working with Hashmap as a consultant to organizations.
Hedge Your Sales Pitch
Protect your career. Never sell a cloud data warehouse as a way to save costs for the organization. Sell a cloud data warehouse as a way to optimize costs around data analytics. Data storage is cheaper relative to the costs of storing data on-prem for almost all organizations. Computation is cheaper relative to on-prem from an apples-to-apples comparison for almost all organizations. The caveat to both of these statements is that the cloud represents near-infinite storage and computes the amount. This can mean that your “$/activity” cost is cheaper, but your “Total Dollars” can be much higher. This brings up the next tip…
Manage Your Warehouse
A major draw for a cloud data warehouse is the ‘set-it-and-forget-it’ mentality. And this is largely true. You are no longer focused on keeping the servers up and running, and you stop losing sleep because your server is running on a system that went off support five years ago.
This does not mean that you can stop managing your data warehouse. It means that your focus needs to shift from keeping the hardware and software up and running 24/7 to monitoring usage and activity (and still keeping some hardware up and running 24/7). I recommend that teams set up monitoring and usage by department or cost center early. Early monitoring provides extraordinary benefits:
- Who is in and who is out. Knowing what areas of the organization are adopting the cloud data warehouse and what areas of the organization continue to use legacy systems can help you build a change management program that meets teams where they are. You can target advanced training to teams and individuals that are embracing the new technology. Conversely, you can identify teams that are not adopting the new warehouse and determine why. Is critical data missing from the cloud? Are there security issues? Are there knowledge/skill gaps preventing adoption? Once identified, you can work to address those problems team by team.
- Enhanced change management. Knowing what teams and individuals are doing in the new data warehouse can allow you to broadcast wins and innovative uses across an organization. A tool designed for the safety team may be a great tool for the business units to improve health and safety in the field or on the factory floor while promoting the cloud data warehouse's adoption.
- Retire legacy systems. Comparing utilization across systems can help you promote and prioritize data or tools as they become available in the cloud. As solutions are put in place in the cloud, you can then begin providing a retirement schedule for teams still using legacy systems. You can also track IT’s success at migrating the organization to the cloud data warehouse based on utilization reports.
Quick Tip: A Data Management program that uses data families to endorse and own their data in the cloud can ensure that you create a single source for the truth early within the cloud data warehouse. When two warehouses disagree, the default for correct data becomes the new data warehouse. This can help build buy-in with teams like Internal Audit, forcing teams to migrate to the new data warehouse.
- Always be optimizing. Educate teams on optimizing data use early on. New technologies require new mindsets and a cloud data warehouse is no different. Teams need to learn how to optimize their queries and their Business Intelligence tools. They need to remember that each query comes at a cost. If a quarterly report for accounting is needed, they shouldn’t pull five years of data just because they can. IT needs to take an active role in optimizing data models and queries used across the organization to run as cheaply and efficiently as possible.
- Educate. Many cloud data warehouses come with ways to optimize computational costs. Be sure your teams and power users of data within the business understand what drives costs up and what keeps costs down.
- Be transparent. Even if accounting does not allocate technical costs, you should consider assigning computational costs to respective cost centers. Help teams see what they are doing with their data and (potentially) what that cost is. In an ideal world, you can help teams identify areas of wasteful spending in the cloud and work to reduce those costs. It would be best if you were careful with this approach for two reasons. First, you don’t want to be chasing pennies. If unoptimized queries are costing the IT organization a few hundred dollars, and the overall budget for IT is several million dollars, you probably don’t need to spend significant time on this. You also want to ensure that managers don’t misinterpret your message and halt valuable analytics programs to reduce costs.
- Set up safeguards. When possible, limit the amount of computational power that a query can use or set controls on how long a query can run. Also, many cloud warehouses come with alerts that can be set up to let you know in near-real time if a query is in danger of costing too much to run. Be sure these are in place early!
Move Slow
Slow is smooth, smooth is fast. Take time to do your research. Take time to meet with your team to discuss all of the facets that come with a warehouse (security, accessibility, change management, data management, data governance, etc.). Take time to gain buy-in and identify champions across the organization. Take time to use a cloud migration project as an opportunity to break down needless barriers between teams and the organization’s data. Align your cloud data warehouse strategy to any existing data strategies or programs that might exist. Work to align with any company strategies. This can build buy-in and win the organization's endorsements, but it will help you prioritize data to be migrated and modeled.
Use a Guide
Managing and growing a data warehouse should be a core competency of any healthy IT organization. Standing up to a cloud data warehouse should not be. Don’t hesitate to bring in an outside team, like Hashmap, that has this as a core competency. They can help not only stand up the warehouse (which is deceptively easy), but they can help you configure it in ways to anticipate the future while keeping costs low.
Move Fast
Once you have a plan in place and arrive at the execution stage, consider bringing in additional help to move quickly. Retiring a legacy warehouse can take anywhere from six to twenty-four months. A lot can happen in that time period so consider bringing in outside help to accelerate this timeline. You don’t want to be nine months into your strategy and have the project put on hold because it is making a major acquisition and taking all IT resources for support. You also want to avoid the scenario where you have to maintain two or more data warehouses at once. I have personally had to deal with this because of an acquisition and because we were upgrading the software that our operations teams used. Both created enormous logistical headaches and put key deliverables at risk. In my experience, there is also a point where the broader organization’s buy-in reaches a tipping point, and many teams begin adopting the new warehouse. You want to make sure that the data they need is either in the new warehouse or migrated in the new warehouse in a reasonable amount of time so that you don’t lose that critical buy-in.
Ready to Accelerate Your Digital Transformation?
At Hashmap, we work with our clients to build better together.
If you are considering moving data and analytics products and applications to the cloud or if you would like help and guidance and a few best practices in delivering higher value outcomes in your existing cloud program, then please contact us.
Hashmap, an NTT DATA Company, offers a range of enablement workshops and assessment services, cloud modernization and migration services, and consulting service packages as part of our data and cloud service offerings. We would be glad to work through your specific requirements.
Hashmap’s Data & Cloud Migration and Modernization Workshop is an interactive, two-hour experience for you and your team to help understand how to accelerate desired outcomes, reduce risk, and enable modern data readiness. We’ll talk through options and make sure that everyone has a good understanding of what should be prioritized, typical project phases, and how to mitigate risk. Sign up today for our complimentary workshop.
Feel free to share on other channels and be sure and keep up with all new content from Hashmap, an NTT Data Company, here. To listen in on a casual conversation about all things data engineering and the cloud, check out Hashmap’s podcast Hashmap on Tap as well on Spotify, Apple, Google, and other popular streaming apps.
Other Tools and Content You Might Like
Scottie Bryan is a Delivery Manager with Hashmap, an NTT Data Company, providing Data, Cloud, IoT, and AI/ML solutions and consulting expertise across industries with a group of innovative technologists and domain experts accelerating high-value business outcomes for our customers. Connect with Scottie on LinkedIn.
