Azure Storage Cost Optimization Part II: Design Principles & Best Practices
Introduction: Want to optimize the storage-account cost without any trade-off with performance/operational excellence? This article gives a deep-dive into design considerations, configuration checklists and recommended options w.r.t Azure Data Lake Storage Gen 2, so that the end user is able to cut any un-useful costs on the storage-account thus ensures to meet a sweet balance between the expected operations with an affordable budget.
Main Checklist
- Consider cost savings by reserving data capacity (Azure Reserved Capacity)
- Organize data into access tiers
- Use lifecycle policy to move data between access tiers
- Optimizing by analyzing the monthly trends of various KPI’s/Metrics of storage-account monitoring section
- Storage-account cost break-down by analyzing various operations on it to spotlight the things in action and adpating the design principles based on the observed trend
- Setup custom in-house solution to have dynamic cost analysis & alerts (Dynamic Cost Analysis & Alerts (DCAA) in Azure Cost Management)
Considerations for Cost-Optimization & Operational-Excellence
- Consider cost savings by reserving data capacity for block blob storage (e.g., when there is surety for 2–3 years of utility)
- Organize data into access tiers (frequently accessed data in hot < less accessed data in cool/archive tier)
- Use lifecycle policy to move data between access tiers (Lifecycle management policies can be created to move data in various tiers e.g., move data to archive tier when its not modified in 90 days or so)
- Periodically dispose and clean up unused storage resources (such as old snapshots, old blob versions etc.)
- Consider Azure Blob access time tracking and access time-based lifecycle management (by setting up a policy based on last access time to: cost-effective backup storage options)
- Transition your data from a hotter access tier to a cooler access tier if there’s no access for a period
- Delete your data if there’s no access for an extended period (hot → cool → archive → Purge)
Note: For initial design-schema, the cost based on resource infrastructure & expected operations, can be predicted via Azure Price Calculator
Design, Plan and Manage Costs
- First, estimate costs by using the Azure pricing calculator. After you create your storage account, optimize the account so that you pay only for what you need
- Use cost management features to set budgets, alerts and monitor costs. Also review forecasted costs, and monitor spending trends to identify areas where you might want to act
- Also check Azure Advisor for operational optimization w.r.t Azure-Resources used.
- Keep in mind that costs for Blob Storage are only a portion of the monthly costs in your Azure bill. You can apply similar methods to manage costs for all the Azure services
- Data storage and metadata are billed per GB on a monthly basis.
When you create or use Blob Storage resources, you’ll be charged for the following meters:
[Note]
- Data traffic might also incur networking costs. See the Bandwidth pricing
- Link to Azure Data Lake Storage pricing
- Link to Azure Blob Storage pricing
Use index-tags for blob-lifecycle-management features
For detailed Info. Click on the Complete Article
General Storage-Account Design-Considerations / Setting-Configurations
As mentioned in the above sections based on the business requirements the design & configuration of storage-account can be done for both the cost optimization and operational excellence. So lets explore Azure Portal to get full insights!!
- Look at the ‘Overview’ section to know the properties of the storage-account
2. Explore the Lifecycle Management (LCM) rules for the storage account to save some costs like chaging the access-tier of the data based on the frequency of interaction with it i.e., Hot → Cool → Archive → Delete
3. Also, automation based tasks can be set-up e.g., to get monthly cost of the resource to get a median-trend to analyze and further improve the costs
4. Further, costs can be optimized by analyzing the monthly trends of various KPI’s of storage-account monitoring section e.g., insights, workbooks & activity logs with custom-based metric as per the requirement. And thus, utilizing these trend charts to modify or adapt much suitable design considerations for the storage-account
e.g.,
- Performance Data
- Availability Data
- Capacity Data
- Operational Data via Custom-Query
Example use-case of storage-account cost & billing break-down by analyzing various operations on it
Below is the graph I get from cost analysis breakdown for a dummy storage-account:
The parameters filled in the cost-management dashboard are as under:
- storage-account: <storage account name>
- granularity: accumulated
- time-frame: last-month
- group-by: Meter
Further to check in depth each type of transaction as mentioned in the above graphical-image, do the following:
- Go to the storage account > Metrics
- Select Transactions as the metric
- Click Add Filter and select API Name as the property
- Select the API names you want to explore in detail
Note: Another option is Diagnostics Logging for the storage type you’re interested in (but no recommended) as if a storage-account gets lot of traffic than lot of logs will be written, and You are charged the same rates for reads, writes and storage for this option.
