Cloud Didn’t Kill Storage
But it did just lose your data
At a big IT event, I asked, “If you care about storage management, raise your hand.” I saw a smattering of hands.
Next, “Put your hand down if you aren’t a storage admin.”
One hand stayed in the air.
I asked the lone holdout, “Why do you care about storage management?”
“What? No. I need a raffle ticket. There’s a raffle at the end of this talk, right?”
Storage management is the least appreciated part of IT infrastructure… which is saying something. Business users understand server and network issues. Security and backup teams overwhelm them with graphic horror stories. The storage team just gets a budget cut and requirements to store twice as much data.
What is Storage Management?
When you buy storage, you have to consider (at least) five factors:
- Capacity — How much data do I want to store?
- Durability — How much do I not want to lose my data?
- Availability — How long can I go without being able to read or write my data?
- Performance — How fast do I need to get at my data?
- Cost — How much am I willing to pay?
Different products optimize for different factors. That’s why you see hundreds of storage products on the market. It’s why you see individual vendors sell dozens of storage products. There is no one-size-fits-all product.
Storage management is meeting the storage needs of all the different business applications.
Storage management is also knowing that you’ll always fall short.
Doesn’t Cloud Make Storage Management Go Away?
No.
The same five factors still matter to your applications, even in the cloud.
That’s why cloud providers offer different types of storage. AWS alone offers:
- Local Storage (3 types)– Hard Drive, Flash, NVMe Flash
- Block Storage (4 types)– IOPS Flash (io1), General Purpose Flash (gp2), Throughput Optimized Hard Drive (st1), and Cold Hard Drive (sc1)
- Object Storage (4 types)– Standard S3, Standard Infrequently Accessed S3, One-Zone Infrequently Accessed S3, Glacier
That’s 11 types of storage for one cloud. Has your head started to spin?
It gets worse. You face all the old storage management challenges plus some new ones.
Wait, Cloud Makes Storage Management MORE Important?
Yes.
Cost Overruns — Overprovisioning
It’s easy for application teams to run up a massive cloud storage bill.
On-premises environments built up checks-and-balances. The application team asks the storage management team for storage resources. The storage team makes them justify the request. The storage management team needs to buy more hardware. Purchasing makes them justify the request. The process slows down the application team, but it prevents reckless consumption and business surprises.
Cloud environments wipe away the checks-and-balances. Application owners pick a type of cloud storage for their application. The cloud provider tells them how much performance they get for each GB of capacity. They see that it costs a dime or less a month. They don’t have to ask anyone for approval. So they buy a big pool of storage. Why not? Turn on the faucet. It’s cheap.
Then they need more capacity. So they buy more. Turn on the faucet again. And it’s cheap.
Then they need more performance. So they buy more. Turn on the faucet again and again and again. And it’s still cheap.
The bill for 100s of TBs of storage comes in at the end of the month. Nobody turned off the faucet. They’ve run up the water bill and flooded the house.
It’s not cheap anymore.
Cost Overruns — Storage Silos
It’s easy for application teams to waste money in the cloud.
Twenty year ago, each server had its own storage. Server A used 1% of its storage. Server B used 100% of its storage and ran out. Server B couldn’t use Server A’s storage. Then storage teams adopted shared storage, SAN or NAS. Now they can give storage resources to any application or system that needs it. Shared storage eliminates the waste from the islands of server storage.
Today, cloud environments don’t share storage.
When I create a cloud instance (aka server), I buy storage for it.
When I create a second cloud instance, I buy storage for it.
When I create a third cloud instance, I buy storage for it!
When I … you get the idea.
We’ve re-created the “island of storage” problem. Except at cloud scale, application teams end up with island chains of storage. Even Larry Ellison doesn’t have archipelago money.
Data Loss — Trusting the Wrong Type of Storage
It’s easy for application teams to lose their data in the cloud.
On-premises environments use resilient shared storage systems. Application owners don’t think about RAID, mirroring, checksums, and data consistency tools. That’s because the storage management team does.
In the cloud, picking storage is like a “Choose Your Own Adventure” story, where you always lose:
- You choose local storage. When the node goes away, so does your storage. One day you shut down the node to save money. You lost all your data. Start over.
- You choose block storage. AWS states, “Amazon EBS volumes are designed for an annual failure rate (AFR) of between 0.1% — 0.2%, where failure refers to a complete or partial loss of the volume.” At some point this year you’ll lose a volume. You lost an application’s data. Start over.
- You choose S3 object storage. You chose resilient storage! You win! **
** Your application can’t run with such slow performance. You lost your customers. Start over.
NOTE: Backups can help. But you should have resilient production storage and backups. Backups should be your last resort, not your first option.
When Should I Plan for Cloud Storage Management?
Now.
It’s time to start thinking about storage management in the cloud.
Today, you’re running cloud applications that don’t need to store data. Or that don’t care if they lose data. Or you’ve just been lucky.
The time to manage cloud storage is coming. You need to start planning now. It can save your applications, your business and your career.
Oh, and don’t spend time worrying about the raffle. They’re always rigged for the customer with the biggest deal pending, anyway.
