Python for Azure: Dynamic Cost Analysis & Alerts (DCAA) in Azure Cost Management
Introduction: In the well-architected framework (WAF) terminology of cloud adoption framework (CAF), “Cost Optimization” is also one of the vital pillar among the other five which are “Operational Excellence”, “Performance Efficiency”, “Reliability” and “Security”.
“Azure Budgets” is a fantastic tool for organizing and enforcing responsibility for Azure costs. Azure Budgets let you proactively control costs by allowing you to define a threshold number for your Azure infrastructure. Using Azure Action Groups, user can add alerts to the budgets. At the level of a subscription or resource group, a budget can be made.
For more information on Azure cost management for setting up budgets and alerts refer to the official Microsoft documentation.
However, the limitation of this cost management via “budgets and alerts” is that the values for them is a hard-coded value i.e., a static value for actual, forecasted or custom-set threshold value.
There is also a forecasted projection provided via Azure cost management but that is again based on “historical trends”, the main challenge here is to have “dynamic monitoring and alerts” for some unexpected, random and sudden high Azure resource “utilization spike” that can increases the cost of Azure consumption drastically.
In order to tackle this challenge, I have developed a custom solution via Python SDK for Azure especially using “Cost Management API” to make this cost management either for a list of “Resource Groups” or “Subscriptions” dynamic; so that some sudden high utilization abnormalities of Azure resources can be first identified and later handled via condition based alerting mechanism. In this workflow demo threshold-value is set to +20% i.e., if the current cost is 20% more than the total-cost of the cost incurred for the last-week, then the script will generate an exception with message, ‘“Cost Exceeded! Take Action!”
Hands-On Implementation via Azure Portal & Python SDK for Azure
Prerequisites
- Python 3.6 or later is required to use this package
- You must have an Azure subscription to run the python code based workflow.
- Its assumed there is already a “Resource Group” (RG) or several RG deployed under the subscription and there exists operational “Resources” within the RG so that we can extract some corresponding cost-expenditures.
- Also each RG has a tag so that in the scenario I as listed below specific tags can be used for the target RG.
Setup
- Clone or download this project repository: Python-for-Azure
- Open the project folder in Visual Studio Code or your IDE of choice.
- From the root location of the project folder run the following command to install all the requirements in particular Azure related libraries for Python with pip command:
pip install -r requirements.txtWorkflow
1. The script below demonstrates the usage of Python SDK for Azure for implementing the above said workflow i.e., setting up dynamic cost analysis and alert (DCAA) for the two scenarios as listed below:
- Scenario I: Setting up DCAA for specific resource-groups under a similar or different subscription [Python Script]
- Scenario II: Setting up DCAA for a subscription or a list of subscriptions that we want to monitor [Python Script]
2. In the terminal of the IDE do the following steps:
- Log in to your Azure account
az login --tenant <tenant_id>[Info]: Now, the “_get_credential” method using “DefaultAzureCredential” library can do the authentication properly.
3. After selecting the correct ‘Python Interpreter’ & correct ‘Configuration’ for the scope of your project like “Working Directory” etc. , run the scripts in the directory “dynamic_cost_analysis_alerts/src/” for both the Scenarios I & II
4. The structure of the DCAA project repository is as under:
- The ‘pipelines’ folder contains the DCAA pipeline .yml file
- The ‘result’ folder container the tabular-result of both the scenario I & II
- The ‘src’ folder contains the source-code i.e., the python-script for both the scenarios
- The ‘utility’ folder contains the .csv file for scenario I & II respectively. Here in these files for scenario I all the resource-groups to be monitored under the same or different subscriptions can be entered; similarly for scenario II all thee different subscriptions with their name can be entered, so that the python-script can read the values from these .csv files
Scenario I - DCAA for specific resource-groups under a subscription
Scenario II — DCAA for a subscription or a list of subscriptions
Verifying Scenario II cost-estimation at the subscription-level directly from the Azure portal “cost analysis” section of the targeted subscription.
Alerting Mechanism for Dynamic Cost Analysis & Alerts
The alerting mechanism for DCAA works as mentioned below:
Case I: The pipeline runs daily, as per the threshold value (THV) set in the python script i.e., +20%; if the current cost of scenario I or II is less than the THV → then the Python script passes → which in turn means the scheduled DCAA pipeline passes → therefore not alert or notification is created, just the result-file gets updated for the records.
Case II: The pipeline runs daily, as per the threshold value (THV) set in the python script i.e., +20%; if the current cost of scenario I or II is greater than the THV → then the Python script fails → which in turn means the scheduled DCAA pipeline fails → therefore alert or notification is created i.e., an email to the specific members/use-case group is sent. Also the result-file gets updated for the records.
The DCAA pipeline for the above two scenarios is published below:
For the “Case II” the notification-configuration has to be set for the “DCAA pipeline” for targeting the specific members/use-case group responsible for Azure related billing/expenditures. Also, the “filter-criteria” has to be set for the pipeline especially for “build-status == failed” so that this alerting mechanism can function properly.
Below an example snippet (in Azure DevOps) is taken where I deployed the DCAA customized solution for my Azure Infrastructure.
So if the cost limit for Scenario I & II increases, and now with the ‘alerting mechanism’ as configured above, the registered users for this event will get an email as shown below:
NOTE: For double verification, in the initial setup do compare the cost results of the DCAA pipeline with the one shown in the Cost-Management of the Azure Portal with the same filter e.g. total-cost parameter, this can assure correctness/accuracy from the very beginning.
