Why is Storage Management So Painful?
or What Do All Those Storage Admins Do?
If you want to understand the vastness of data, don’t measure it in petabytes. Measure the number of people managing your storage.
In 2015, I walked into a customer’s New Jersey IT center. Their head of storage pointed at an endless sea of cubes and boasted, “That’s our storage team area.” The VP of Infrastructure (a networking guy) asked in amazement, “What do all these people do?” The head of storage chuckled (in that storage person way), “We keep the business running.”
My conclusion:
- If that many people work on it, storage management is a really big deal.
- If that many people work on it, storage management is really broken.
Why is storage management so painful and expensive? Why do we need all those people and what do they do all day?
Cost, Performance, Capacity — Pick 2
Imagine a storage continuum. On one end is “fast”. On the other is “big”. Each application wants a different point on that continuum. To make it worse, in the real world that continuum is multi-dimensional. There isn’t just one way to define “performance” and one way to define “capacity”.
Cost vs. Performance
Real World Performance Challenge: Applications define “performance” differently.
Some need storage to respond quickly to a transaction (e.g. a payroll application). Others want to do many transactions at the same time (e.g. online purchasing system). Still others want to process oceans of data (e.g. data analytics).
Real World Performance Answer: Many types of storage arrays.
Vendors offer Tier-1, All-Flash, Hybrid Disk-Flash, Scale-Out NAS, Deduplication, and other types of arrays because each type of storage solves a different performance challenge.
Storage teams need people with special expertise to manage each type of array.
Cost vs. Capacity
Real World Capacity Challenge: Companies don’t want to pay to store unimportant data.
Everything generates data, and nobody deletes any of it. It piles up like garbage during a Parisian sanitation strike. Why waste the high-performance storage on old data? Move it somewhere cheaper.
Real World Capacity Answer: Many Types of Tiering
Some businesses move pieces of data (e.g. Carl in Accounting’s vacation photos from Decorah, Iowa) to cheaper storage. Others move whole applications (e.g. the performance review application we stopped using in 2012). Some tier storage to cloud. Some tier to other arrays. Some still tier to tape.
Storage teams need people with special expertise to manage tiering.
Storage teams balance requirements for performance, capacity, and cost. With a lower budget every quarter.
Availability vs. Cost
Storage hardware breaks. Components get old, power surges, or disasters hit the site. Sometimes, the hardware fails. Other times, it returns incorrect data.
Unlike servers, you can’t “just get a new one” because that hardware holds your data. You either need to have a copy of it elsewhere or figure out how to get it back from the broken device.
Storage systems use error-correction codes, RAID, mirroring, and other reliability techniques. Some protect you from more failures than others. Those usually cost more.
Storage teams need to people with special expertise to manage different availability configurations.
Storage Teams Plan, Users Laugh
Then users come in.
All application owners want from storage is:
- Performance specific to their application’s needs
- Capacity specific to their application’s needs
- At the lowest cost
- Without any errors or downtime
But they don’t know:
- How much capacity they’ll need.
- How much performance they’ll need.
Still, let’s pretend that the storage team has done the impossible. They built a stable, cost-optimized storage environment that meets all their users’ needs!
Then the requirements change because:
- Business priorities changed
- Government regulations changed
- The application itself changed
- An executive ate a blueberry muffin instead of a blueberry bagel for breakfast.
The Road to Hell is Paved with Storage Migration
All storage administrators end up in the land of storage migration. It hurts worse than rubbing your eyes while chopping ghost peppers. We all hate the “application downtime for maintenance over the weekend” email. Guess what? The storage admin, working all weekend to move all the data, hates it a lot more than you.
As one storage admin said, “I start planning to migrate an application the day I deploy it.”
Storage teams need people to maintain the environment.
Can We Solve This Problem?
With shrinking budgets, companies and admins can’t survive managing storage this way. It’s too slow, too complex, and too expensive.
In the last decade, we have simplified storage management:
- All-Flash Arrays– These arrays reduce the effort to manage storage performance. Despite the marketing, All-Flash cannot handle all workloads at the best cost
- Hyperconverged– With flash storage inside the server, you can eliminate the storage management. Everything is “good enough”. It works well in smaller deployments.
- Data Analytics — Analytics (e.g. Nimble’s Infosight) can recommend optimizations.
- Software-Defined — Storage admins can change storage configurations more easily.
We’ve been making storage simpler, but it’s not simple, yet.
When storage can meet the users’ changing requirements without administrator involvement, then we shall be free.
Until then, storage management is like sprinting on a treadmill covered in Legos. You can’t win; you just try to not lose the will to keep running.
